Psychological state recognition method based on multi-modal biological feature and scale data fusion

Abstract

This application provides a method for identifying psychological states based on the fusion of multimodal biometrics and scale data, belonging to the field of psychological state recognition technology. The method collects physiological data from target subjects when answering psychological scales and extracts the physiological features corresponding to each question; calculates initial scale scores based on the answer data and determines the initial correlations between questions; constructs a heterogeneous graph model with questions as nodes, physiological features and semantic features as node features, and the initial correlations as connecting edges; inputs this model into a graph attention network, dynamically adjusts the attention coefficients through the fusion values ​​of physiological and semantic features, and iteratively aggregates neighbor node information to obtain higher-order node representations; after graph pooling the higher-order node representations to output a graph representation, it fuses it with the initial scale scores across modalities, and determines the psychological state based on the fusion result. This application integrates physiological responses and scale semantic information to achieve accurate psychological state recognition.

Description

Technical Field

[0001] This application relates to the field of mental state recognition technology, and in particular to a mental state recognition method based on the fusion of multimodal biometrics and scale data. Background Technology

[0002] Mental state recognition has broad application prospects in clinical diagnosis, mental health management, and human-computer interaction. By analyzing an individual's behavioral responses and physiological changes, it can provide a basis for objective assessment of mental state, thereby enabling early warning and personalized intervention.

[0003] In existing technologies, assessment methods based on psychological scales are the mainstream approach for identifying psychological states. The specific process of this mainstream approach involves collecting textual responses from test subjects, weighting and summing the scores for each dimension of the questions according to pre-designed scoring rules to obtain a total score reflecting the psychological state, and then determining the test subject's psychological state by comparing the total score with a preset threshold. By collecting individuals' textual responses to scale questions and calculating scores for each dimension according to a pre-designed scoring model, their psychological state can be quantified.

[0004] However, the above methods rely solely on single textual responses, failing to capture the implicit physiological response patterns of individuals during the response process. This makes the evaluation results susceptible to subjective factors and response biases. Furthermore, traditional weighted scoring methods treat the contributions of each question as independent, making it difficult to characterize the deep semantic connections and interactions between questions. This limits the evaluation model's ability to finely distinguish individual psychological states. At the same time, the lack of a unified modeling framework for multi-source heterogeneous information such as textual and physiological data makes it difficult to utilize them collaboratively within the same model. Therefore, existing technologies suffer from insufficient multi-source information fusion and limited recognition accuracy. Summary of the Invention

[0005] The purpose of this application is to provide a psychological state recognition method based on the fusion of multimodal biometrics and scale data, so as to solve the problems of insufficient utilization of multi-source information fusion and limited recognition accuracy in the existing technology.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a method for identifying psychological states based on the fusion of multimodal biometrics and scale data, comprising: Collect physiological data of the target subjects when they answer the psychological scale, and extract the physiological characteristics corresponding to each item in the psychological scale; Calculate initial scale scores based on the target object's answer data and determine the initial correlation between items; A heterogeneous graph model is constructed using the question as a node, the physiological features and the semantic features of the question as node features, and the initial association relationship as connecting edges. The heterogeneous graph model is input into the graph attention network, which adjusts the attention coefficient based on the fusion value of the physiological features of neighboring nodes and the semantic features of the current node, and aggregates the features of neighboring nodes according to the attention coefficient to obtain the high-order node representation of each node. The higher-order node representations are then subjected to graph pooling to output a graph representation. The graphical representation is fused with the initial scale score across modalities, and the psychological state of the target object is determined based on the fusion result.

[0007] Based on the above scheme, this application synchronously collects physiological data and extracts the physiological features corresponding to each item, enabling physiological responses to participate in the construction of a heterogeneous graph model as node features; it uses the initial correlation between items as the basis for connecting edges in the heterogeneous graph model, allowing the correlation information to be reflected in the graph structure; and it performs cross-modal fusion of the initial scale scores with the graph representation output by graph pooling in the final stage, organically combining the traditional quantitative result of scale scores with graph structure features. Thus, the three types of basic information—physiological features, initial correlations, and initial scale scores—are all fully utilized in subsequent steps, resulting in a closed-loop overall process and avoiding the problem of isolated features.

[0008] As one possible implementation method, the extraction of physiological features corresponding to each item in the psychological scale includes: recording the temporal relationship between the items of the psychological scale and the physiological data; based on the temporal relationship, extracting the physiological data between the presentation time of each item and the completion time of the answer as a physiological data segment corresponding to that item; calculating at least one of the mean, peak value, or variation range for each physiological data segment as a statistic, and using the statistic as the physiological response intensity feature of the corresponding item. Based on this method, the physiological features are aligned with the timing of item completion, accurately reflecting the physiological arousal level of the target subject when answering each item.

[0009] As one possible implementation method, the step of calculating the initial scale score based on the target object's answer data and determining the initial correlation between questions includes: based on a preset structured model, determining the individual score of each question from the answer data, accumulating the scores according to the evaluation dimensions to which each question belongs to obtain the dimension scores, and weighting and summing them according to the contribution proportion of each evaluation dimension to obtain the initial scale score; marking the correlation strength between any two questions belonging to the same evaluation dimension as a first preset value, and marking the correlation strength between any two questions belonging to different evaluation dimensions as a second preset value, wherein the first preset value is greater than the second preset value; and summing the correlation strengths between all questions to constitute the initial correlation between questions. The preset structured model includes a first preset data table, a second preset data table, and a third preset data table; the first preset data table stores the attribution relationship between the psychological scale and each evaluation dimension, as well as the contribution proportion of each evaluation dimension; the second preset data table stores the question identifier, question text content, and the identifier of the evaluation dimension to which the question belongs in the psychological scale; and the third preset data table stores the correspondence between the question identifier and the question score. Based on this method, the correlation between scale scores and items based on assessment dimensions can be obtained simultaneously based on a pre-set structured model, providing two types of basic information for subsequent mapping and cross-modal fusion.

[0010] As one possible implementation method, the construction of the heterogeneous graph model includes: defining each item in the psychological scale as a node; for each node, inputting the text content of the item corresponding to the node into a preset semantic encoding model to obtain the semantic features of the node; determining the association strength between the node and all other nodes based on the initial association relationship to form the topological structure features of the node; extracting the physiological response intensity features of the item corresponding to the node from the physiological features; concatenating the semantic features, the topological structure features, and the physiological response intensity features to form the combined features of the node; for any two nodes, determining the association strength between their corresponding items based on the initial association relationship, and establishing a connection edge between them if the association strength is greater than zero; combining all nodes, all connection edges, and the combined features of each node to construct the heterogeneous graph model. Based on this method, semantic, association, and physiological heterogeneous information are integrated into the same graph structure, laying the foundation for the collaborative processing of multimodal features.

[0011] As one possible implementation method, the processing of the graph attention network includes: using the combined features of each node as input features; for each current node, determining its neighboring nodes; summing the components of the semantic feature vector of the current node to obtain a semantic scalar, and multiplying it by the physiological response intensity features of each neighboring node to obtain a fusion value; performing a normalized exponential transformation on the fusion values ​​of all neighboring nodes to obtain attention weights; weighting and summing the combined features of the neighboring nodes according to the attention weights and adding them to the combined features of the current node at corresponding positions to obtain a first-order representation; repeating the above steps with the first-order representation as a new input until a preset number of iterations is reached to obtain a higher-order node representation. Based on this method, the iterative graph attention mechanism enables the model to focus on question interaction patterns with close physiological-semantic connections.

[0012] As one possible implementation, the graph pooling of the higher-order node representations to output a graph representation includes: determining the evaluation dimension to which each node belongs based on the structured model; constructing a graph pooling layer containing a learnable projection module; for each evaluation dimension, selecting the higher-order node representations of all nodes belonging to that evaluation dimension from the higher-order node representations, and calculating the pooling score of each node through the projection module; sorting all nodes belonging to the same evaluation dimension in descending order according to the pooling scores of each node, and selecting the top-ranked nodes as retained nodes according to a preset retention ratio; aggregating the higher-order node representations of the retained nodes by average according to their corresponding positions to generate a dimensional graph representation for the corresponding evaluation dimension; and summarizing the dimensional graph representations of all evaluation dimensions as the output graph representation. Since the higher-order node representations of all nodes come from the same graph attention network and have the same feature dimensions, and are aggregated by averaging according to their corresponding positions, the dimensional graph representations output for each evaluation dimension have the same dimensions. Based on this method, key node information is retained by grouping by dimension and the dimensional consistency of the dimensional graph representations is ensured, providing conditions for subsequent unified fusion with the score vector.

[0013] As one possible implementation method, the cross-modal fusion and psychological state determination process includes: using the initial scale score as a scalar value, copying and expanding it according to the dimensions represented by the dimensional graph to obtain a score vector; for each evaluation dimension's dimensional graph representation, calculating the element-wise product between the score vector and the dimensional graph representation as a first fusion component, and the element-wise weighted sum as a second fusion component, concatenating the two components to form the fusion feature vector corresponding to that evaluation dimension; concatenating the fusion feature vectors of all evaluation dimensions to obtain the overall fusion feature; inputting the overall fusion feature into a preset evaluation model to obtain a psychological state index; comparing the psychological state index with the index intervals corresponding to multiple preset psychological state levels to determine the psychological state of the target object. Based on this method, the fusion of traditional scale scores and multimodal graph features through various interactive methods improves the comprehensiveness and accuracy of the recognition results.

[0014] As one possible implementation method, the graph attention network, the projection module in the graph pooling layer, and the evaluation model are obtained through end-to-end joint training, thereby enabling the parameters of each module to be optimized in a coordinated manner, further improving the accuracy of mental state recognition.

[0015] Secondly, this application also provides a psychological state recognition system based on the fusion of multimodal biometrics and scale data, including a data acquisition module, a calculation module, a construction module, a generation module, a processing module, and a determination module, which are respectively used to perform the corresponding steps of the above method.

[0016] Thirdly, this application also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the steps of the above-described method.

[0017] Based on the above scheme, this application has the following beneficial effects compared with the prior art: First, by collecting physiological data during the scale response and extracting the physiological features corresponding to each question, the correlation between physiological signals and answering behavior is realized, improving the objectivity of psychological state assessment; Second, by constructing a heterogeneous graph model with questions as nodes and integrating physiological and semantic features, multi-source heterogeneous information is incorporated into a unified modeling framework; Third, by introducing a graph attention network and dynamically adjusting the attention weights based on the fusion values ​​of physiological and semantic features, the model can focus on question interaction patterns with close physiological-semantic correlation; Fourth, through graph pooling and cross-modal fusion mechanisms, the three types of basic information—physiological features, initial correlation, and initial scale scores—are fully utilized in subsequent steps, avoiding the problem of isolated features, and the overall process logic is closed, resulting in a significant improvement in recognition accuracy compared with the traditional weighted scoring method. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a psychological state recognition method based on the fusion of multimodal biometrics and scale data, provided for an embodiment of this application; Figure 2 A schematic diagram illustrating the calculation process for initial scale scores and initial correlations between items, provided in an embodiment of this application; Figure 3 A schematic diagram of the graph attention network iterative aggregation process provided in the embodiments of this application; Figure 4 A schematic diagram of the cross-modal fusion processing flow provided in the embodiments of this application; Figure 5 This is a schematic diagram of the heterogeneous graph model structure provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of a psychological state recognition system based on the fusion of multimodal biometrics and scale data, provided for an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0021] To address the shortcomings of existing technologies in utilizing multi-source information fusion and limiting recognition accuracy, this application proposes a psychological state recognition method based on the fusion of multimodal biometrics and scale data. This method simultaneously collects physiological and question-response data during scale completion, constructing a heterogeneous graph model with questions as nodes. The physiological and semantic features of the questions, as well as the relationships between questions, are uniformly incorporated into the graph structure. A graph attention network dynamically fuses physiological and semantic features to generate high-order node representations, which are then fused cross-modally with the initial scale scores to output a comprehensive psychological state assessment result.

[0022] A flowchart illustrating the method provided in this application embodiment is shown below. Figure 1 As shown, the method includes steps S101 to S106.

[0023] S101. Collect physiological data of the target subject when answering the psychological scale, and extract the physiological characteristics corresponding to each item in the psychological scale.

[0024] Physiological data refers to signals collected by physiological sensors during the target subject's response to psychological scales, including but not limited to electrophysiological signals that reflect an individual's emotions and stress state, such as heart rate, skin conductance, and respiratory rate. Physiological characteristics are quantitative indicators extracted from physiological data that characterize the intensity of the target subject's physiological response when answering specific questions.

[0025] Since existing methods rely solely on textual response data and cannot reflect the real-time physiological arousal level of the test subjects during the response, this step lays the data foundation for subsequent multimodal fusion by synchronously collecting physiological data and establishing alignment relationships according to the question time relationship.

[0026] In one implementation, S101 specifically includes steps S1011 to S1012.

[0027] S1011. Record the temporal relationship between the items of the psychological scale and the physiological data.

[0028] Among them, the time relationship refers to the correspondence between the presentation time of each question, the answering time, and the physiological data collection time.

[0029] In step S1011, the physiological data acquisition device is first activated, and the questions of the psychological scale are presented simultaneously. The time when each question is first presented, the time when the target subject completes the answer, and the continuous timestamps recorded by the physiological data acquisition device are recorded to establish a time correspondence between the questions and the physiological data.

[0030] S1012. Based on the time relationship, extract the physiological data between the time each question is presented and the time the answer is completed as the physiological data segment corresponding to the question; calculate at least one of the mean, peak value or change range for each physiological data segment as a statistic, and use the statistic as the physiological response intensity feature of the corresponding question.

[0031] Physiological data segments refer to data segments extracted from continuous physiological signals that correspond to the time period of answering a particular question. Physiological response intensity features are scalar values ​​extracted from physiological data segments that characterize the intensity of the target object's physiological response when answering the question.

[0032] As an example, suppose a psychological scale contains three items, Items 1 through 3. Item 1 is presented at second 0, and the target subject completes the response at second 15; Item 2 is presented at second 15, and the subject completes the response at second 30; Item 3 is presented at second 30, and the subject completes the response at second 45. Simultaneously, a physiological data acquisition device continuously records heart rate signals starting from second 0. For Item 1, the corresponding time period is from second 0 to second 15. A data segment is extracted from the heart rate signal for this time period, and the average heart rate within this segment is calculated as the physiological response intensity characteristic. Assuming the calculated average heart rate is 78 beats per minute, this value is taken as the physiological response intensity characteristic for Item 1; similarly, the physiological response intensity characteristic for Item 2 is 82 beats per minute; and the physiological response intensity characteristic for Item 3 is 76 beats per minute.

[0033] S102. Calculate the initial scale score based on the target object's answer data, and determine the initial correlation between the items.

[0034] The answer data refers to the target subject's response records on the psychological scale, including the question identifier for each question and the selected answer option. The initial scale scores will be fused with the graph representation in S106 across modalities and will be used in the final determination of psychological state; the initial association relationships will be used in S103 as the basis for establishing the connection edges of the heterogeneous graph model.

[0035] In one implementation, step S102 is based on a preset structured model. The preset structured model includes a first preset data table, a second preset data table, and a third preset data table. The first preset data table stores the attribution relationship between the psychological scale and each assessment dimension, as well as the contribution weight of each assessment dimension; the second preset data table stores the item identifiers, item text content, and the identifiers of the assessment dimensions to which the items belong in the psychological scale; the third preset data table stores the correspondence between the item identifiers and item scores. These data tables are pre-filled and fixed in the system by the psychological scale designer according to the scale's structure and scoring rules, and can be configured according to the type of psychological scale actually used.

[0036] like Figure 2 As shown, S102 specifically includes steps S1021 to S1024.

[0037] S1021. Based on the answer data of the target object, retrieve the question score corresponding to each question from the third preset data table, and combine it with the answer options of each question in the answer data to determine the single question score of the target object on each question.

[0038] Here, the question score refers to the preset base score weight for each question; the answer options refer to the alternative answers selected by the target object for each question; and the single question score refers to the actual score obtained by the target object for a single question. In this embodiment of the application, for each question, firstly, the base score weight of the question is retrieved from the third preset data table according to the question identifier; then, according to the target object's answer options, the corresponding option score is determined; and the single question score is obtained by multiplying the base score weight by the option score.

[0039] As an example, suppose the third preset data table stores the base score weight of question 1 as 5, and the answer options for question 1 include four options: A, B, C, and D, with corresponding option scores of 4, 3, 2, and 1 respectively; if the target selects option B for question 1, the single question score is the product of the base score weight of 5 and the option score of 3, which is 15 points.

[0040] S1022. Based on the first preset data table and the second preset data table, determine all questions belonging to the same evaluation dimension, and sum the individual scores of all questions belonging to the same evaluation dimension to obtain the dimension score of each evaluation dimension.

[0041] In this context, assessment dimensions refer to different psychological trait levels measured by the psychological scale, such as anxiety dimensions and depression dimensions; dimension scores refer to the sum of all item scores for the target subject on a certain assessment dimension. As an example, assume the assessment dimensions include anxiety and depression dimensions. The second pre-set data table shows that items 1 and 2 belong to the anxiety dimension, and item 3 belongs to the depression dimension; adding the single-item score of 15 for item 1 to the single-item score of 12 for item 2 yields a dimension score of 27 for the anxiety dimension; the single-item score of 18 for item 3 yields a dimension score of 18 for the depression dimension.

[0042] S1023. Based on the contribution ratio of each evaluation dimension in the final result in the first preset data table, the dimension scores of each evaluation dimension are weighted and summed to obtain the initial scale score.

[0043] The contribution weight refers to the weight coefficient of each assessment dimension in the final scale score, and the sum of the contribution weights of all assessment dimensions is 1. As an example, assuming the contribution weight of the anxiety dimension is 0.6, the contribution weight of the depression dimension is 0.4, the dimension score of the anxiety dimension is 27, and the dimension score of the depression dimension is 18; then the initial scale score is 27 multiplied by 0.6 plus 18 multiplied by 0.4, which equals 23.4. This initial scale score will serve as one of the inputs for cross-modal fusion in subsequent step S106.

[0044] S1024. Identify all questions belonging to the same evaluation dimension based on the first preset data table and the second preset data table. Mark the correlation strength between any two questions belonging to the same evaluation dimension as a first preset value. Mark the correlation strength between any two questions belonging to different evaluation dimensions as a second preset value. The first preset value is greater than the second preset value. Summarize the correlation strength between all questions to form the initial correlation relationship between questions.

[0045] The initial association relationship refers to a matrix or set describing the strength of the association between any two items in the psychological scale. A first preset value is used to characterize a strong semantic association between items within the same dimension, while a second preset value is used to characterize no obvious semantic association between items in different dimensions. For example, the first preset value can be set to 1, and the second preset value can be set to 0. Specific values ​​can be adjusted according to actual needs. This initial association relationship will be used in subsequent step S103 to establish the connection edges in the heterogeneous graph model.

[0046] As an example, the first preset value is set to 1, and the second preset value is set to 0; if both question 1 and question 2 belong to the anxiety dimension, then the correlation strength between them is 1; if question 1 and question 3 belong to different dimensions, then the correlation strength between them is 0; if question 2 and question 3 belong to different dimensions, then the correlation strength between them is 0.

[0047] S103. Construct a heterogeneous graph model using the question as a node, the physiological features and the semantic features of the question as node features, and the initial association relationship as connecting edges.

[0048] Among them, the heterogeneous graph model is a graph structure data model in which nodes contain multiple types of features. In this application, the nodes in the graph represent questions, and the connections between nodes indicate the relationships between questions. Each node contains three types of features: semantic features, topological features, and physiological response intensity features, hence the name heterogeneous graph model.

[0049] Since S102 obtained the initial correlation between the questions and S101 obtained the physiological characteristics corresponding to each question, it is necessary to organize the above heterogeneous information and the semantic information of the questions themselves in a unified manner so that they can be processed by graph neural networks in the future. Therefore, this step constructs a heterogeneous graph model with questions as nodes.

[0050] As one implementation method, S103 specifically includes steps S1031 to S1035.

[0051] S1031. Define each item in the psychological scale as a node, and all nodes constitute a node set.

[0052] In this embodiment of the application, all items of the psychological scale are traversed, and a corresponding node is created for each item. For example, if the psychological scale contains three items: item 1, item 2, and item 3, then node 1, node 2, and node 3 are created.

[0053] S1032. For each node, input the text content of the question corresponding to the node into a preset semantic encoding model to obtain the semantic features of the node; determine the correlation strength between the node and all other nodes according to the initial correlation to form the topological structure features of the node; extract the physiological response intensity features of the question corresponding to the node from the physiological features.

[0054] Semantic features refer to the vector form obtained by converting the text content of the question through a pre-defined semantic encoding model, used to represent the semantic information of the question text. Topological structure features refer to the vector composed of the association strength between a node and all other nodes, used to represent the positional relationship of the node in the graph structure. Physiological response intensity features are specific numerical values ​​corresponding to specific questions from the physiological features extracted from S1012.

[0055] The pre-defined semantic encoding model can be a BERT model based on the Transformer architecture. Its construction process is as follows: a large-scale general corpus is used as training data, and masked language modeling and next sentence prediction are used as training objectives. The pre-trained semantic encoding model is obtained through self-supervised pre-training. The pre-trained model can be further fine-tuned based on a small-scale labeled corpus in the field of psychological scales. After the pre-training is completed, the question text is input into the model, and the vector corresponding to the special marker position at the beginning of the sentence in the output of the last layer of the model is taken as the fixed-dimensional semantic feature vector of the question.

[0056] As an example, for question 1 corresponding to node 1, its text content is "I feel nervous and uneasy." This is converted into 3-dimensional vectors 0.2, 0.5, and 0.3 as semantic features using a semantic encoding model. The association strength between node 1 and nodes 2 and 3 is extracted from the initial association matrix, resulting in vectors 1 and 0 as topological structure features. The average heart rate of 78 beats per minute corresponding to question 1 is extracted from the physiological response intensity features. It should be noted that in practical applications, the dimension of semantic features is usually much higher than 3, for example, it could be 768 dimensions. This example uses a low-dimensional dimension only for illustrative purposes.

[0057] S1033. The semantic features, topological features, and physiological response intensity features of each node are concatenated to form the combined features of the node.

[0058] The concatenation refers to joining multiple vectors sequentially end-to-end to form a vector with a higher dimension. As an example, for node 1, the semantic features are vectors 0.2, 0.5, and 0.3, the topological features are vectors 1 and 0, and the physiological response intensity feature is the value 78. The combined features of node 1 are then vectors 0.2, 0.5, 0.3, 1, 0, and 78. Since the semantic features, topological features, and physiological response intensity features of all nodes have the same dimension, the combined features of each node also have the same dimension.

[0059] S1034. For any two nodes, determine the association strength between the corresponding questions based on the initial association relationship. If the association strength is greater than zero, establish a connection edge between the two nodes. All connection edges constitute an edge set.

[0060] As an example, for node 1 and node 2, the corresponding value in the initial association matrix is ​​1, which is greater than 0, so a connection edge is established between node 1 and node 2; for node 1 and node 3, the corresponding value is 0, so no connection edge is established; for node 2 and node 3, the corresponding value is 0, so no connection edge is established.

[0061] S1035. Combine the node set, the edge set, and the combined features of each node to construct the heterogeneous graph model.

[0062] As an example, the node set (nodes 1, 2, and 3) and the edge set (the edge between nodes 1 and 2) are combined, and a combination feature is added to each node to construct a heterogeneous graph model containing three nodes and one edge. The combination feature of node 1 is the vector 0.2, 0.5, 0.3, 1, 0, 78. A schematic diagram of the constructed heterogeneous graph model can be found in [reference needed]. Figure 5 .

[0063] S104. Input the heterogeneous graph model into the graph attention network. The graph attention network adjusts the attention coefficient based on the fusion value of the physiological features of neighboring nodes and the semantic features of the current node. Based on the attention coefficient, the features of neighboring nodes are aggregated to obtain the high-order node representation of each node.

[0064] Graph attention networks are a type of graph neural network that dynamically calculates attention weights between nodes based on node features, thus assigning different importance to different neighbors when aggregating neighbor node information. Higher-order node representations refer to the node features generated after multiple layers of iterative aggregation.

[0065] The construction process of the graph attention network used in this application is as follows: the network contains multiple graph attention layers, each containing a linear mapping module for feature transformation and an attention coefficient calculation module; the network uses physiological data, answer data, and labeled psychological state indices of multiple sets of target objects collected in history as training data, and forms an end-to-end overall network with subsequent graph pooling layers and evaluation models; during training, the training objective is to minimize the mean square error between the psychological state index predicted by the network and the labeled results, and the network parameters are optimized using a stochastic gradient descent algorithm based on backpropagation until the loss function converges or the preset number of training rounds is reached.

[0066] In one implementation, S104 specifically includes steps S1041 to S1045.

[0067] S1041. Use the combined features of each node in the heterogeneous graph model as the input features of the graph attention network.

[0068] S1042. For each current node, determine all neighboring nodes that have a connection edge with the current node.

[0069] For example, for node 1, the node that has a connecting edge to it is node 2, so the neighbor node of node 1 is node 2.

[0070] S1043. Summing the components of the semantic feature vector of the current node to obtain the semantic scalar of the current node, multiplying the semantic scalar by the physiological response intensity feature of each neighboring node to obtain the fusion value between the current node and the neighboring node; normalizing the fusion value of all neighboring nodes corresponding to the current node by performing a normalized exponential transformation to obtain the attention weight between the current node and each neighboring node.

[0071] Here, the fusion value is a scalar determined by the semantic information of the current node and the physiological response intensity information of its neighboring nodes. Attention weights are the importance coefficients assigned to each neighboring node, with the sum of all neighboring node attention weights equal to 1. Normalization exponential transformation is the operation that transforms a set of values ​​into positive values ​​and sums them to 1.

[0072] As an example, for node 1 and its neighbor node 2, the semantic features of node 1 are vectors 0.2, 0.5, and 0.3. The sum of the components of this vector is a semantic scalar of 1.0. The physiological response intensity feature of node 2 is 82. Multiplying the two together gives a fusion value of 1.0 multiplied by 82, which is 82. Since node 1 has only one neighbor node, the attention weight after normalization is 1.

[0073] S1044. Based on the attention weight, the combined features of each neighboring node are weighted and summed to obtain the aggregated feature, and the aggregated feature is added to the combined feature of the current node at the corresponding positions to obtain the first-order representation of the current node.

[0074] As an example, for node 1, the combined features of its neighbor node 2 are vectors 0.6, 0.2, 0.2, 1, 0, 82, with an attention weight of 1, and the aggregated features are vectors 0.6, 0.2, 0.2, 1, 0, 82. This aggregated feature is added to node 1's own combined feature vectors 0.2, 0.5, 0.3, 1, 0, 78 at corresponding positions to obtain the first-order representation of node 1 as vectors 0.8, 0.7, 0.5, 2, 0, 160.

[0075] S1045. Using the first-order representation as a new input feature, repeatedly execute the steps of determining neighboring nodes, calculating attention weights, weighted summation and adding to the corresponding position until a preset number of iterations is reached, and use the final representation as the higher-order node representation of each node.

[0076] The preset number of iterations refers to the number of layers in which the graph attention network performs multi-layer aggregation. Each iteration represents one hop in the outward propagation of information. The specific value can be set according to the size of the graph and the actual recognition accuracy requirements, for example, 2 to 4. Figure 3 The specific implementation process of steps S1041 to S1045 is described in general.

[0077] S105. Perform graph pooling on the higher-order node representation and output the graph representation.

[0078] Graph pooling refers to the operation of transforming node-level representations in a graph structure into graph-level or subgraph-level representations. Since the higher-order node representations obtained in S104 are node-level features, while subsequent cross-modal fusion requires graph-level or subgraph-level features, this step completes the transformation from node-level features to graph-level features by constructing a graph pooling layer.

[0079] In one implementation, this step performs group pooling based on the evaluation dimension to which each node corresponds to the question. Specifically, the evaluation dimension to which each node corresponds to the question is determined according to the identifier of the evaluation dimension stored in the second preset data table in the structured model; a graph pooling layer containing a learnable projection module is constructed, wherein the projection module is composed of trainable projection vectors, and the pooling score is obtained by performing an inner product operation between the node feature matrix and the projection vector and taking the norm; the graph pooling layer is jointly trained end-to-end with the graph attention network and the evaluation model.

[0080] S1051. Determine the evaluation dimension to which each node corresponds to the question based on the structured model.

[0081] S1052. For each evaluation dimension, select all high-order node representations belonging to that evaluation dimension from the high-order node representations, and stack the selected high-order node representations to form a node feature matrix; input the node feature matrix to the projection module, and the projection module calculates a projection value for each node in the node feature matrix as the pooling score of the corresponding node.

[0082] S1053. Based on the pooling scores of each node, sort all nodes belonging to the same evaluation dimension in descending order, and select the top-ranked nodes from the sorting results as retained nodes according to a preset retention ratio. The preset retention ratio can be set to, for example, 0.5, and the specific value can be adjusted according to the number of nodes in each dimension and actual needs.

[0083] S1054. Aggregate the higher-order node representations of the retained nodes by their corresponding positions to generate a dimensional graph representation for the corresponding evaluation dimension; summarize the dimensional graph representations of all evaluation dimensions as the graph representation output.

[0084] It should be noted that since the higher-order node representations of all nodes output in S104 come from the same graph attention network and have the same feature dimensions, and since this step aggregates the retained nodes by averaging their corresponding positions, the dimensional graph representations output by each evaluation dimension have the same dimension. This dimensional consistency ensures that in S106, the single scalar score can be expanded into a score vector of uniform length and fused with the dimensional graph representations.

[0085] As an example, assume the assessment dimensions include an anxiety dimension and a depression dimension. The anxiety dimension contains nodes 1 and 2, and the depression dimension contains node 3. The preset retention ratio is 0.5. The projection module calculates the pooling score for each node. The pooling score for node 1 is 0.9, for node 2 it is 0.6, and for node 3 it is 0.8. For the anxiety dimension, the node with the highest pooling score between nodes 1 and 2 is node 1. Node 1 is retained, and its higher-order node representation is used as the dimension graph representation of the anxiety dimension. For the depression dimension, since there is only one node 3, all nodes are retained, and its higher-order node representation is used as the dimension graph representation of the depression dimension.

[0086] S106. The graphical representation is fused with the initial scale score across modalities, and the psychological state of the target object is determined based on the fusion result.

[0087] Since the graph representation obtained in S105 contains multimodal fusion features extracted from the graph structure, and the initial scale score obtained in S102 contains traditional scale rating information, both reflect the psychological state of the test subject from different modalities. This step explicitly introduces the initial scale score into the cross-modal fusion process, allowing it to play a role in the final judgment and avoiding treating the score as an isolated feature.

[0088] As one implementation method, such as Figure 4 As shown, S106 specifically includes steps S1061 to S1065.

[0089] S1061. Obtain the dimensional graph representation of each evaluation dimension, where each dimensional graph is represented as a feature vector.

[0090] For example, the dimension diagram for the anxiety dimension is represented as vectors 0.8, 0.7, 0.5, 2, 0, 160, and the dimension diagram for the depression dimension is represented as vectors 0.1, 0.1, 0.8, 0, 1, 76. Both have a dimension of 6.

[0091] S1062. The initial scale score is used as a scalar value and copied and expanded according to the dimensions represented by the dimension diagram to obtain a score vector with the same dimensions as those represented by the dimension diagram.

[0092] Obtain the initial scale score from S1023; repeatedly expand this scalar value into a vector such that the dimension of the vector is the same as the dimension represented by the dimension diagram. For example, if the initial scale score is 23.4 and the dimension diagram has a dimension of 6, then 23.4 will be expanded into the vector 23.4, 23.4, 23.4, 23.4, 23.4, 23.4.

[0093] S1063. For the dimension graph representation of each evaluation dimension, calculate the element-wise product between the score vector and the dimension graph representation as the first fusion component, and the element-wise weighted sum between the score vector and the dimension graph representation as the second fusion component. Concatenate the first fusion component and the second fusion component to form the fusion feature vector corresponding to the evaluation dimension. Concatenate the fusion feature vectors of all evaluation dimensions to obtain the overall fusion feature.

[0094] Element-wise multiplication refers to multiplying the elements at the same positions of two vectors to obtain a new vector. Element-wise weighted sum refers to multiplying the elements at the same positions of two vectors by preset weighting coefficients and then adding them together to obtain a new vector.

[0095] As an example, for the anxiety dimension, the score vector is vector 23.4, 23.4, 23.4, 23.4, 23.4, 23.4, and the dimension diagram is represented as vector 0.8, 0.7, 0.5, 2, 0, 160. The first fusion component is obtained by element-wise multiplication, which is vector 18.72, 16.38, 11.7, 46.8, 0, 3744. The weighting coefficient of the element-wise weighted sum is taken as 0.5, so the second fusion component is calculated according to the corresponding position, which is vector 12.1, 12.05, 11.95, 12.7, 11.7, 91.7. The first fusion component and the second fusion component are concatenated end to end to obtain the fusion feature vector corresponding to the anxiety dimension. Similarly, the fusion feature vector corresponding to the depression dimension is calculated. The fusion feature vectors of the two assessment dimensions are concatenated to form the overall fusion feature.

[0096] S1064. Input the overall fusion features into the preset evaluation model, and the evaluation model outputs the psychological state index.

[0097] The psychological state index refers to the numerical value output by the model used to characterize the severity of psychological state. The preset evaluation model adopts a multilayer perceptron structure containing multiple fully connected layers and activation function layers, with a fully connected layer at the end with an output dimension of 1 to output the comprehensive index. The graph representation output by the graph pooling layer and the overall fusion feature after cross-modal fusion of the initial scale scores are used as input, and the manually labeled psychological state index is used as the supervision signal. The mean squared error is used as the loss function, and the network parameters in the evaluation model are iteratively optimized through a gradient descent algorithm based on backpropagation until the loss function converges or the preset number of training rounds is reached.

[0098] For example, if the overall fusion features are input into the evaluation model and the model outputs a value of 75.3, then the psychological state index is 75.3.

[0099] S1065. The psychological state index is compared with the index intervals corresponding to multiple preset psychological state levels, and the psychological state level corresponding to the index interval to which the psychological state index belongs is determined as the psychological state of the target object.

[0100] As an example, the preset psychological state levels and their corresponding index ranges are: normal state corresponds to 0 to 40, mild abnormality corresponds to 40 to 70, moderate abnormality corresponds to 70 to 90, and severe abnormality corresponds to 90 to 100; the psychological state index is 75.3, which falls within the 70 to 90 range, so the target's psychological state is determined to be moderately abnormal.

[0101] In one implementation, the graph attention network, the projection module in the graph pooling layer, and the evaluation model are jointly trained end-to-end. The specific training process is as follows: Physiological data, answer data, and labeled psychological state indices of multiple historically collected target objects are used as training data; each set of training data is processed according to the method described in this application to obtain the predicted psychological state index output by the evaluation model; the mean squared error between the predicted psychological state index and the corresponding labeled psychological state index is used as the loss function, and a stochastic gradient descent algorithm based on backpropagation is used to iteratively optimize the parameters of the graph attention network, the projection module, and the evaluation model until the loss function converges or reaches a preset number of training rounds.

[0102] Based on the above method, this application also provides a psychological state recognition system based on the fusion of multimodal biometrics and scale data, referring to... Figure 6 The system includes: a data acquisition module 61, a calculation module 62, a construction module 63, a generation module 64, a processing module 65, and a determination module 66, which are used to execute the corresponding steps of the above method. For specific implementation details, please refer to the aforementioned method embodiment section.

[0103] Specifically, the data acquisition module 61 is used to collect physiological data of the target object when answering the psychological scale, and to extract the physiological characteristics corresponding to each item in the psychological scale. The calculation module 62 is used to calculate the initial scale score based on the target object's answer data and determine the initial correlation between the questions; Module 63 is used to construct a heterogeneous graph model with the question as a node, the physiological features and the semantic features of the question as node features, and the initial association relationship as connecting edges. The generation module 64 is used to input the heterogeneous graph model into the graph attention network to obtain the high-order node representations of each node; Processing module 65 is used to perform graph pooling on the higher-order node representation and output a graph representation; The determination module 66 is used to perform cross-modal fusion of the graphical representation and the initial scale score, and determine the psychological state of the target object based on the fusion result.

[0104] This application also provides an electronic device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program to implement the steps of the above-described method for identifying mental states based on the fusion of multimodal biometrics and scale data.

[0105] The foregoing has provided a detailed description of a psychological state recognition method based on the fusion of multimodal biometrics and scale data, as provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for identifying psychological states based on the fusion of multimodal biometrics and scale data, characterized in that, include: Collect physiological data of the target subjects when they answer the psychological scale, and extract the physiological characteristics corresponding to each item in the psychological scale; Calculate initial scale scores based on the target object's answer data and determine the initial correlation between items; A heterogeneous graph model is constructed using the question as a node, the physiological features and the semantic features of the question as node features, and the initial association relationship as connecting edges. The heterogeneous graph model is input into the graph attention network, which adjusts the attention coefficient based on the fusion value of the physiological features of neighboring nodes and the semantic features of the current node, and aggregates the features of neighboring nodes according to the attention coefficient to obtain the high-order node representation of each node. The higher-order node representations are then subjected to graph pooling to output a graph representation. The graphical representation is fused with the initial scale score across modalities, and the psychological state of the target object is determined based on the fusion result.

2. The method according to claim 1, characterized in that, The extraction of physiological characteristics corresponding to each item in the psychological scale includes: Record the temporal relationship between the items of the psychological scale and the physiological data, including the correspondence between the presentation time of each item, the answer time, and the physiological data collection time; Based on the aforementioned time relationship, the physiological data between the time each question is presented and the time the answer is completed is extracted as the physiological data segment corresponding to that question; For each physiological data segment, at least one of the mean, peak value, or range of change is calculated as a statistic, and the statistic is used as a physiological response intensity feature for the corresponding question.

3. The method according to claim 1, characterized in that, The step of calculating initial scale scores based on the target object's answer data and determining initial correlations between items includes: Based on a pre-set structured model, the individual score of each question is determined from the answer data, the dimension scores are accumulated according to the evaluation dimensions to which each question belongs, and the initial scale score is obtained by weighted summation according to the contribution ratio of each evaluation dimension. The correlation strength between any two questions belonging to the same evaluation dimension is marked as a first preset value, and the correlation strength between any two questions belonging to different evaluation dimensions is marked as a second preset value, wherein the first preset value is greater than the second preset value; the correlation strength between all questions is summarized to form the initial correlation relationship between the questions; The preset structured model includes a first preset data table, a second preset data table, and a third preset data table. The first preset data table stores the attribution relationship between the psychological scale and each assessment dimension, as well as the contribution ratio of each assessment dimension. The second preset data table stores the item identifier, item text content, and the identifier of the assessment dimension to which the item belongs in the psychological scale. The third preset data table stores the correspondence between the item identifier and the item score.

4. The method according to claim 1, characterized in that, The construction of the heterogeneous graph model includes: Each item in the psychological scale is defined as a node; For each node, the text content of the question corresponding to the node is input into a preset semantic encoding model to obtain the semantic features of the node; The association strength between the node and all other nodes is determined based on the initial association relationship, constituting the topological structure feature of the node; the physiological response intensity feature of the question corresponding to the node is extracted from the physiological features; the semantic features, the topological structure feature and the physiological response intensity feature are concatenated to form the combined feature of the node; For any two nodes, determine the association strength between their corresponding questions based on the initial association relationship. If the association strength is greater than zero, establish a connection edge between them. The heterogeneous graph model is constructed by combining all nodes, all connecting edges, and the combined features of each node.

5. The method according to claim 4, characterized in that, The process involves inputting the heterogeneous graph model into a graph attention network, where the graph attention network adjusts the attention coefficients based on the fusion value of the physiological features of neighboring nodes and the semantic features of the current node, and aggregates the features of neighboring nodes according to the attention coefficients to obtain the higher-order node representations of each node, including: The combined features of each node in the heterogeneous graph model are used as the input features of the graph attention network; For each current node, identify all neighboring nodes that have connecting edges with the current node; sum the components of the semantic feature vector of the current node to obtain the semantic scalar of the current node; multiply the semantic scalar by the physiological response intensity feature of each neighboring node to obtain the fusion value between the current node and the neighboring node; perform a normalized exponential transformation on the fusion value of all neighboring nodes corresponding to the current node to obtain the attention weight between the current node and each neighboring node. Based on the attention weights, the combined features of each neighboring node are weighted and summed to obtain the aggregated features. The aggregated features are then added to the combined features of the current node at corresponding positions to obtain the first-order representation of the current node. The first-order representation is used as a new input feature. The steps of determining neighboring nodes, calculating attention weights, weighted summation and adding to the corresponding position are repeated until a preset number of iterations are reached. The final representation is used as the higher-order node representation of each node.

6. The method according to claim 3, characterized in that, The step of performing graph pooling on the higher-order node representation to output a graph representation includes: The evaluation dimension to which each node corresponds to a question is determined based on the structured model. Construct a graph pooling layer that includes a learnable projection module. For each evaluation dimension, select the higher-order node representations of all nodes belonging to that evaluation dimension from the higher-order node representations, and calculate the pooling score of each node through the projection module. Based on the pooling scores of each node, all nodes belonging to the same evaluation dimension are sorted in descending order, and the nodes with the highest ranking are selected as retained nodes according to the preset retention ratio. The higher-order node representations of the retained nodes are averaged and aggregated according to their corresponding positions to generate the dimensional graph representation of the corresponding evaluation dimension; the dimensional graph representations of all evaluation dimensions are summarized as the graph representation output, and the dimensions of each dimensional graph representation in the graph representation are the same.

7. The method according to claim 6, characterized in that, The step of performing cross-modal fusion of the graphical representation with the initial scale score, and determining the psychological state of the target object based on the fusion result, includes: The initial scale score is used as a scalar value and copied and expanded according to the dimensions represented by the dimension diagram to obtain a score vector with the same dimensions as those represented by the dimension diagram. For each evaluation dimension's dimensional graph representation, the element-wise product between the score vector and the dimensional graph representation is calculated as the first fusion component, and the element-wise weighted sum between the score vector and the dimensional graph representation is calculated as the second fusion component. The first fusion component and the second fusion component are then concatenated to form the fusion feature vector corresponding to the evaluation dimension. The overall fusion feature is obtained by concatenating the fusion feature vectors of all evaluation dimensions. The overall fusion feature is then input into a preset evaluation model, which outputs a psychological state index. The psychological state index is compared with multiple preset index intervals corresponding to psychological state levels, and the psychological state level corresponding to the index interval to which the psychological state index belongs is determined as the psychological state of the target object.

8. The method according to claim 7, characterized in that, The graph attention network, the projection module in the graph pooling layer, and the evaluation model are obtained through end-to-end joint training. The training process includes: The training data consisted of physiological data, answer data, and labeled psychological state indices collected from multiple sets of target subjects in the past. Each set of training data is processed according to the method described in claim 1 to obtain the predicted mental state index output by the evaluation model; Using the mean squared error between the predicted mental state index and the corresponding labeled mental state index as the loss function, the parameters of the graph attention network, the projection module, and the evaluation model are iteratively optimized using a backpropagation-based stochastic gradient descent algorithm until the loss function converges or reaches a preset number of training rounds.

9. A psychological state recognition system based on the fusion of multimodal biometrics and scale data, characterized in that, include: The data acquisition module is used to collect physiological data of the target subject when answering the psychological scale, and to extract the physiological characteristics corresponding to each item in the psychological scale. The calculation module is used to calculate the initial scale score based on the target object's answer data and determine the initial correlation between the questions; The construction module is used to construct a heterogeneous graph model with the question as a node, the physiological features and the semantic features of the question as node features, and the initial association relationship as connecting edges; The generation module is used to input the heterogeneous graph model into the graph attention network to obtain the high-order node representations of each node; The processing module is used to perform graph pooling on the higher-order node representation and output a graph representation; The determination module is used to perform cross-modal fusion of the graphical representation and the initial scale score, and determine the psychological state of the target object based on the fusion result.

10. An electronic device, characterized in that, It includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program to implement the steps of the psychological state recognition method based on the fusion of multimodal biometrics and scale data as described in any one of claims 1 to 8.

Images