A method and device for recognizing depressive mood disorders based on facial behavior data

Abstract

The application provides a depression disorder identification method and device based on facial behavior data, which enables a neural network model to identify whether a subject has a depression disorder through facial behavior data based on any content and form of stimulation data, improves the generalization performance of the neural network model, and has important significance for deploying the model in an actual medical scene. The method comprises the following steps: obtaining target stimulation data for stimulating the senses of a target subject; obtaining target facial behavior data generated by the target subject after being stimulated by the target stimulation data; encoding the target stimulation data to obtain a first vector representation; encoding the target facial behavior data to obtain a second vector representation; inputting the first vector representation and the second vector representation into a neural network model to obtain a classification result output by the model for indicating whether there is a depression disorder by identifying the relationship characteristics between the target stimulation data and the target facial behavior data.

Description

Technical Field

[0001] This application relates to the field of health monitoring technology, and in particular to a method and device for identifying depressive mood disorders based on facial behavior data. Background Technology

[0002] In determining whether a patient has a depressive disorder, psychiatrists often use interviews to determine this, but this method is highly subjective and has a low accuracy rate.

[0003] With the development of artificial intelligence, more and more researchers are using AI technology to seek objective indicators for identifying depressive mood disorders. Using facial behavior data to identify depressive mood disorders holds significant promise in the field of affective computing. Facial behavior data contains a wealth of valuable information regarding emotional expression. Current methods using neural network models to identify depressive mood disorders through facial behavior data, while improving the objectivity and accuracy of identification, suffer from low generalization performance, making them difficult to deploy in real-world medical scenarios. For example, based on stimulus content A, facial behavior data from individuals with depressive mood disorders and healthy individuals are collected separately, forming dataset A. A neural network model trained on dataset A can then learn the differences between individuals with depressive mood disorders and healthy individuals based on their responses. However, in practice, it has been found that the neural network model trained on dataset A also learns features irrelevant to identifying depressive mood disorders, resulting in learned differences including spurious differences, leading to poor generalization performance on dataset B. Dataset B consists of responses from individuals with depressive mood disorders and healthy individuals collected separately based on stimulus content B.

[0004] Because it is impossible to exhaustively collect a dataset of facial behaviors to train neural network models, and it is uncertain what kind of stimulus content should be used to identify depressive mood disorders in real medical scenarios, current neural network models for identifying depressive mood disorders are difficult to deploy in real medical scenarios. Summary of the Invention

[0005] This application provides a method and apparatus for identifying depressive mood disorders based on facial behavior data, enabling neural network models to identify whether subjects have depressive mood disorders based on stimulus data of any content and form through facial behavior data, thereby improving the generalization performance of neural network models and having important significance for deploying models in actual medical scenarios.

[0006] Firstly, a method for identifying depressive mood disorders based on facial behavioral data is provided, characterized by comprising: Acquire target stimulus data provided to target subjects, which is used to stimulate the senses of target subjects. Acquire target facial behavior data generated by the target subjects after being stimulated by the target stimulus data; Encode the target stimulus data to obtain the first vector representation of the emotional features of the target stimulus data; Encode the target facial behavior data to obtain a second vector representation of the facial features of the target facial behavior data; Input the first vector representation and the second vector representation into the Transformer model to obtain the classification result output by the Transformer model by identifying the relationship features between the target stimulus data and the target facial behavior data. The classification result is used to indicate whether the target subject has a depressive mood disorder.

[0007] In one feasible design, the target facial behavior data is video data. The target facial behavior data is encoded to obtain a second vector representation of the facial features, including: After encoding the target facial behavior data using a three-dimensional convolutional neural network, the first intermediate vector representation is obtained; Input the first intermediate vector representation into the Long Short-Term Memory (LSTM) network to obtain the second intermediate vector representation output by the LSTM network. The sum of the first intermediate vector representation and the second intermediate vector representation is input into the self-attention layer to obtain the third intermediate vector representation output by the self-attention layer. The second intermediate vector representation and the third intermediate vector representation are input into the decoder to obtain the second vector representation of the facial features of the target facial behavior data output by the decoder.

[0008] In one feasible design, the target stimulus data is video data. The target stimulus data is encoded to obtain a first vector representation of its emotional features, including: Input the target stimulus data into the three-dimensional residual network to obtain the first vector representation of the emotional features of the target stimulus data output by the three-dimensional residual network.

[0009] In a feasible design, the method also includes: Acquire a source stimulus domain, which includes at least one stimulus data provided to each of a plurality of subjects, the stimulus data being used to stimulate the subjects’ senses. Acquire the source response domain, which includes the corresponding facial behavior data generated by each subject after being stimulated by at least one stimulus data. Obtain a classification label for each participant; the classification label is used to indicate whether the participant has a depressive mood disorder. A stimulus coding model is used to encode each stimulus data in the source stimulus domain to obtain the first vector representation of each stimulus data; A facial behavior coding model is used to encode each facial behavior data in the source response domain, resulting in a second vector representation of each facial behavior data. Using each first vector representation, each corresponding second vector representation, and the corresponding subject's classification label, a Transformer model, a stimulus encoding model, and a facial behavior encoding model are trained. This enables the Transformer model to learn the relationship features between each stimulus data and each corresponding facial behavior data, and to learn the common relationship features of the classification labels corresponding to subjects with depressive mood disorders from the relationship features of each data pair corresponding to subjects with depressive mood disorders. This enables the stimulus encoding model to learn to extract emotional features from the stimulus data, and enables the facial behavior encoding model to extract facial features of subjects with depressive mood disorders from the facial behavior data.

[0010] In a feasible design, the source stimulus domain includes positive, neutral, and negative stimulus data. Positive stimulus data expresses positive emotions, neutral stimulus data has no emotional bias, and negative stimulus data expresses negative emotions.

[0011] In a feasible design, the method also includes: Acquire the target stimulus domain, which includes at least one stimulus data provided to each of the multiple subjects, and each stimulus data in the target stimulus domain is different from each stimulus data in the source stimulus domain. Obtain the target response domain, which includes the corresponding facial behavior data generated by each subject after passing through at least one stimulus data point in the target stimulus domain; Obtain the classification label for each participant; Encode each stimulus data in the target stimulus domain to obtain the first vector representation of each stimulus data in the target stimulus domain; Encode each facial behavior data in the target response domain to obtain a second vector representation of each facial behavior data in the target response domain; The trained Transformer model is evaluated using each first vector representation of the target stimulus domain, the corresponding second vector representation of the target response domain, and the classification labels of the subjects. The evaluation results are then obtained. Improve the Transformer model based on the evaluation results.

[0012] In one feasible design, the method further includes, prior to encoding the target stimulus data, preprocessing the target stimulus data; and / or, Before encoding the target facial behavior data, the method also includes preprocessing the target facial behavior data.

[0013] In one feasible design, the decoder includes at least one first fully connected layer and at least one first activation layer.

[0014] In a feasible design, the three-dimensional residual network comprises multiple residual blocks.

[0015] Secondly, a device for identifying depressive mood disorders based on facial behavioral data is provided, including: The data acquisition module is used to acquire target stimulus data provided to the target subjects, and the target stimulus data is used to stimulate the senses of the target subjects. The data acquisition module is used to acquire the target facial behavior data generated by the target subject after being stimulated by the target stimulus data; The stimulus encoding model is used to encode target stimulus data to obtain a first vector representation of the emotional features of the target stimulus data; A facial behavior encoding model is used to encode target facial behavior data to obtain a second vector representation of the facial features of the target facial behavior data; The depressive mood disorder identification module is used to input the first vector representation and the second vector representation into the Transformer model, and obtain the classification result output by the Transformer model by identifying the relationship features between the target stimulus data and the target facial behavior data. The classification result is used to indicate whether the target subject has a depressive mood disorder.

[0016] In this embodiment, target stimulus data, which stimulates the senses of the target subject, and the subject's response information (i.e., target facial behavior data) are first acquired. The target stimulus data is encoded to obtain a first target vector representation, which transforms the target stimulus data into encoded data (i.e., a vector representation) with a certain structure and meaning, enabling the Transformer model to interpret and understand the target stimulus data. Similarly, the target facial behavior data is encoded into a second target vector representation, enabling the Transformer model to interpret and understand the target facial behavior data. The first and second target vector representations are input into the Transformer model, allowing the Transformer model to identify whether the target subject has a depressive mood disorder by recognizing the relationship characteristics between the target stimulus data and the target facial behavior data. Compared to current methods that only study the differences in facial behavior data between healthy and depressed individuals for depressive mood disorder identification, the Transformer model used in this application integrates information from the target stimulus data and the target facial behavior data, comprehensively understanding the subject's emotional changes from receiving the target stimulus data to responding based on the essential relationship characteristics between the two, thereby accurately identifying whether the subject has a depressive mood disorder. The depressive mood disorder identification method provided in this application can identify depressive mood disorders regardless of the content and form of the target stimulus data, thus enhancing the generalization performance of the Transformer model and having significant implications for deploying the model in real-world medical scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of an example stimulus-response pattern provided in an exemplary embodiment of this application; Figure 2 This is a schematic flowchart illustrating an exemplary embodiment of a depressive mood disorder identification method based on facial behavior data provided in this application. Figure 3 This is a schematic flowchart illustrating yet another example of a depressive mood disorder identification method based on facial behavior data provided in an exemplary embodiment of this application; Figure 4 This is a schematic diagram of an exemplary facial behavior coding model provided in an exemplary embodiment of this application; Figure 5This is a schematic flowchart illustrating a brain emotion processing process provided in an exemplary embodiment of this application; Figure 6 This is a schematic diagram of an example Transformer model provided in an exemplary embodiment of this application; Figure 7 This is a schematic diagram of an exemplary embodiment of the present application for a depressive mood disorder recognition device based on facial behavior data. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] To improve the objectivity and accuracy of identifying depressive mood disorders, neural network models based on facial cues are currently being used. Facial expressions can convey up to 55% of overall emotional information. For example, individuals with depressive mood disorders (referred to as depressed individuals) often exhibit reduced facial expression richness, such as drooping eyes, frowning, downturned corners of the mouth, reduced smiling, and easy crying. Therefore, facial behavioral data can objectively and accurately identify the presence of depressive mood disorders.

[0021] Current methods for depression identification based on facial behavior data analysis identify whether a subject has a depressive disorder by studying the differences in behavioral information produced by healthy and depressed individuals after receiving the same stimulus. However, this method cannot exhaustively collect datasets containing all stimulus content, including facial behavior differences between healthy and depressed individuals, to train the neural network model. Furthermore, in real-world medical scenarios, it is uncertain which stimulus content should be used to identify depressive disorders. Therefore, current neural network models for identifying depressive disorders are difficult to deploy in real-world medical settings. In other words, the models exhibit poor cross-dataset generalization performance and tend to overfit to the training dataset.

[0022] To address the poor generalization performance of the aforementioned depressive mood disorder identification model, this application, inspired by psychological experiments, refers to the cause of the subject's behavior (also known as a behavioral signal) as a "stimulus," and the resulting facial behavior as a "response." This application finds that a "stimulus"-to-response relationship is prevalent in the process of behavioral signal collection (e.g., Figure 1(As shown). Here, "stimulus" can be understood as an instruction given to the subject, such as asking the subject a question, or requiring the subject to watch a video or picture. "Response" can be understood as the subject's behavior in response to the instruction, such as the content of the answer, the subject's posture, and facial behavior, including not only facial behavior. Clearly, there is a relationship between such stimuli and responses, specifically a constraint relationship: a specific stimulus corresponds to a type of response, such as watching a video yielding facial behavior data. We call this constraint relationship between "stimulus" and "response" a "stimulus-response" pattern. By extracting common features from multiple different "stimulus-response" patterns corresponding to depressed individuals, we can eliminate spurious differences in a single pattern to obtain more essential information for identifying depressive mood disorders, thereby enhancing the model's generalization performance. This application focuses on how to improve the generalization performance of a neural network model that identifies depressive mood disorders based on facial behavior data.

[0023] Stimulus-response pattern recognition aims to elucidate the association between stimuli presented to subjects and subsequent responses, revealing patterns of brain activity under different stimuli. Research indicates that the stimulus-response patterns of depressed individuals differ from those of healthy individuals. While depressed individuals can exhibit expressions of joy, anger, sadness, and happiness, their facial expression activity tends to be reduced compared to healthy individuals, and they are more likely to express negative emotions. On an individual level, responses to stimuli also vary depending on the specific characteristics of the stimulus material. Therefore, neural network models relying solely on facial data to identify the presence of depressive mood disorders have limited generality. In contrast, because the stimulus-response patterns of depressed individuals and healthy groups exhibit relative stability, neural network models based on stimulus-response patterns can not only accurately identify the presence of depressive mood disorders but are also unrestricted by the content of the stimulus.

[0024] Therefore, this application provides a method for identifying depressive mood disorders based on facial behavioral data, such as... Figure 2 As shown, the method includes: S110, Obtain target stimulus data provided to the target subjects.

[0025] Among them, the target stimulus data is used to stimulate the senses of the target subjects.

[0026] Target stimulus data refers to instructions given to the target participant, such as questions asked of the participant, videos or pictures requested for the participant to watch, etc. This application does not limit the form and content of target stimulus data; that is, target stimulus data is a type of multimodal data.

[0027] S120, acquire the target facial behavior data generated by the target subject after being stimulated by the target stimulus data.

[0028] For example, the type of target facial behavior data is image data or video data, etc.

[0029] S130, the target stimulus data is encoded to obtain the first vector representation of the emotional features of the target stimulus data.

[0030] In a feasible design, the target stimulus data is preprocessed before encoding. Preprocessing includes, but is not limited to, filtering, cropping, and downsampling.

[0031] The filtering process is used to select video frames that contain faces, are of good quality, and have no obstructions on the faces.

[0032] The cropping process involves performing face recognition on video frames and then cropping out the target region based on the region of the face. The target region includes the region of the face.

[0033] Downsampling is used to reduce the resolution of video frames, thereby improving the speed of subsequent processing of video frames.

[0034] In a feasible design, such as Figure 3 The facial behavior data encoding section shown uses video data as the target facial behavior data. The encoding of the target facial behavior data is achieved in the following way, resulting in a second vector representation of the facial features of the target facial behavior data: After encoding the target facial behavior data using a three-dimensional convolutional neural network, the first intermediate vector representation is obtained; Input the first intermediate vector representation into the Long Short-Term Memory (LSTM) network to obtain the second intermediate vector representation output by the LSTM network. The sum of the first intermediate vector representation and the second intermediate vector representation is input into the self-attention layer to obtain the third intermediate vector representation output by the self-attention layer. The second intermediate vector representation and the third intermediate vector representation are input into the decoder to obtain the second vector representation of the facial features of the target facial behavior data output by the decoder.

[0035] A facial behavior encoding model is constructed using 3D Convolutional Neural Networks (3DCNN), Long Short-Term Memory (LSTM), a self-attention layer, and a decoder. This application names the facial behavior encoding model "3DCNN-LSTM-SelfAttention," abbreviated as "CLS." The structure of the facial behavior encoding model is as follows: Figure 4 As shown.

[0036] The decoder includes at least one first fully connected layer and at least one first activation layer. For example... Figure 4 As shown, the decoder includes two first fully connected layers and two first activation layers. The connection relationships between the layers are shown in [reference needed]. Figure 4 .

[0037] The 3D convolutional neural network part includes at least two 3D convolutional layers, two second activation layers, two 3D max pooling layers, a 3D average pooling layer, a second fully connected layer, and a third activation layer. The connection relationships between these layers are detailed in [link to documentation]. Figure 4 .

[0038] This application draws inspiration from the physiological basis of cognitive models of brain emotion processing and proposes the CLS model, which is designed to solve the problem of effectively representing topic information.

[0039] Figure 5 These are the main stages of emotion processing in cognitive models of brain emotion processing proposed by some experts. For example... Figure 5 As shown, the core stages of emotion perception and processing after the stimulus is presented include (1) consciously or unconsciously identifying and evaluating the meaning of the stimulus; (2) generating emotional states, emotional expressions and behavioral responses; and (3) upward or downward regulation of emotional states and the identification process.

[0040] During the generation of emotions, the brain's sensory systems (such as vision and hearing) receive information related to stimuli and then transmit it to emotional centers such as the amygdala and prefrontal cortex to assess its emotional significance. This assessment involves feedback mechanisms, through which the brain processes and modulates the emotions associated with stimuli based on prior experiences and internal states. Once the emotional value of a stimulus has been assessed, the brain adjusts the emotional experience through the emotional feedback system, thereby regulating the intensity, duration, and expression of the emotion. Exploring and understanding the feedback mechanisms in emotion processing can provide insights into the mechanisms of emotion regulation and modulation. Therefore, the CLS model in this application is constructed based on the feedback mechanisms observed in emotion processing and aims to capture and represent complex dynamic information.

[0041] In the above method, a three-dimensional convolutional neural network is used to analyze video footage of the subject's facial behavior. Extracting temporal facial information (i.e., the first intermediate vector representation). Then, the extracted temporal facial information... The inputs are fed into the Long Short-Term Memory (LSTM) network and the Self-Attention (SA) layer, respectively, for emotion regulation and expression. The STM network's regulatory output acts on the SA and the first fully connected layer. The detailed process is shown in equations (1), (2), (3), (4), and (5), where C(·) represents the number of channels in the input video data, D(·) represents the depth of the input video data, H(·) represents the height of the input video data, W(·) represents the width of the input video data, T(·) represents the sequence length, and d(·) represents the feature dimension. Formula (1); Formula (2); Formula (3); Formula (4); Formula (5); in, , The video clips showing the facial expressions of the subjects being filmed. This indicates the dimensional information of a video clip, including parameters such as the number of channels, depth, height, and width.

[0042] This represents the output of the 3DCNN. In Extracted temporal facial information The depth and feature dimensions.

[0043] This represents the output of the LSTM (i.e., the second intermediate vector representation). In , They are respectively The depth and feature dimensions.

[0044] and The sum (ADD indicates summation). In , They are respectively The depth and feature dimensions.

[0045] This represents the output of Attention (i.e., the third intermediate vector). In , They are respectively The depth and feature dimensions, among which, , .

[0046] This represents the normalization exponent (softmax) function. , , , Describes the dimension of K. This represents the scaling factor, and T represents the transpose. , , middle , , These are the module parameters in the self-attention layer mechanism.

[0047] This represents the extracted facial feature information of the subject (i.e., the second vector representation). Indicates decoder, In , They are respectively The depth and feature dimensions.

[0048] In the previous example, a deep learning model was used to implement the corresponding functions based on the stages of emotional processing in the brain. Specifically, a three-dimensional convolutional neural network was used to recognize and evaluate visual information, a self-attention layer was used to promote the generation and expression of emotional states, and a long short-term memory network was used to regulate emotional states and the recognition process upwards or downwards. The facial behavior coding model can accurately reflect the sequential flow of emotional processing stages, thereby accurately calculating the facial behavior feature information of the subject and transforming it into encoded data with a certain structure and meaning, enabling the subsequent Transformer model to interpret and understand the subject's facial behavior feature information.

[0049] S140, the target facial behavior data is encoded to obtain the second vector representation of the facial features of the target facial behavior data.

[0050] In one feasible design, the target facial behavior data is preprocessed before encoding. The details of the preprocessing are described in the section on preprocessing target stimulus data, and will not be repeated here.

[0051] In a feasible design, such as Figure 3 The target stimulus data encoding section shown uses video data. The target stimulus data is encoded in the following way to obtain the first vector representation of the emotional features of the target stimulus data: Input the target stimulus data into the three-dimensional residual network to obtain the first vector representation of the emotional features of the target stimulus data output by the three-dimensional residual network.

[0052] Emotional characteristics refer to the emotions expressed in the stimulus data, including positive emotions, negative emotions, and unbiased emotions. Positive emotions include happiness, joy, and excitement, while negative emotions include sadness and grief.

[0053] For example, a three-dimensional residual network includes multiple residual blocks.

[0054] In this application, the 3D Residual Network (3D ResNet) can also serve as the stimulus encoding model.

[0055] For example, this application uses 3D-ResNet18 to process stimulus videos. (i.e., target stimulus data) is encoded to represent emotional characteristics. (i.e., the first vector representation). 3D-ResNet18 consists of multiple residual blocks, and each residual process can be represented by the following formula (6): Formula (6); in, This represents the input to the Residual Block. Represents the residual function. This indicates the output of this Residual Block. This represents the activation function. These represent the parameters of the Residual Block. The entire 3D-ResNet network can be defined as 3DRes, representing the emotional feature information of the entire stimulus data. The extraction process can be represented by the following formula (7): Formula (7); in, Indicates a stimulus video sequence segment, In the equation, C, D, H, and W represent the number of channels, depth, height of the video frame, and width of the video frame, respectively.

[0056] This indicates the emotional information extracted by this module. In They are respectively The depth and feature dimensions.

[0057] In the example above, the use of a three-dimensional residual network can accurately extract the emotional features expressed in the video and transform them into encoded data with a certain structure and meaning, enabling the subsequent Transformer model to interpret and understand the emotional features expressed in the video.

[0058] S150, input the first vector representation and the second vector representation into the Transformer model to obtain the classification result output by the Transformer model by recognizing the relationship features between the target stimulus data and the target facial behavior data.

[0059] The classification result is used to indicate whether the target subject has a depressive mood disorder. The classification result is the predicted classification label.

[0060] For a two-input Transformer model, such as Figure 6 As shown, it consists of at least one sub-block, and each sub-block includes two normalization layers, a multi-head attention layer, a normalization layer, and a feedforward layer. The connection relationships between the layers are shown in [reference needed]. Figure 6 This will not be elaborated upon here.

[0061] Suppose the first vector is represented as The second vector is represented as ,from arrive Relationship Fitting As shown in the following formula (8): Formula (8); in, In These represent the sequence length and feature dimension, respectively, as indicated by the first vector.

[0062] In These represent the sequence length and feature dimension, respectively, as indicated by the second vector.

[0063] For the output of the multi-head attention component, In These refer to sequence length and feature dimension, respectively. It should be understood that in this application... This represents the dimension of the vector representation.

[0064] Let T denote the normalized exponential function, and T denote the transpose.

[0065] , , , , , .

[0066] , , These are the module parameters in the multi-head attention layer. In They are respectively Sequence length and feature dimensions, In They are respectively and The sequence length and feature dimensions. express Dimensions This represents the scaling factor.

[0067] Figure 6 In This is the intermediate feature vector representation used as input to the normalization layer, calculated based on the first and second vector representations.

[0068] Multi-head attention, a key innovation introduced into the Transformer architecture, plays a crucial role in sentiment analysis. This mechanism, by calculating a weighted sum based on the similarity between inputs, allows the model to selectively focus on the most relevant aspects of the input data, thereby enhancing its ability to capture key contextual dependencies and make predictions.

[0069] In the field of emotion analysis, when watching stimuli, individuals from depressed and healthy groups exhibit different emotional and behavioral responses despite experiencing the same stimuli, highlighting significant inter-group differences. The root cause of these differences can be attributed to variations in emotion regulation feedback mechanisms. From a cognitive perspective, the brain receives stimuli and processes them to produce facial expressions. These stimuli and facial behaviors act as the brain's input and output, respectively. This application employs a Transformer architecture to simulate the brain's cognitive processing and function, utilizing the brain's inputs and outputs to learn this processing method. Based on this premise, the Transformer model is trained by combining stimulus data and subject information (including the subjects' facial behavior data and their corresponding classification labels). This application treats each stimulus video as a stimulus domain and each facial behavior data point as a response domain, using the Transformer model to compute stimulus-response patterns between these two domains. This novel approach enables the model to capture and utilize the interaction between stimulus and response domains, thereby promoting a more comprehensive understanding of the emotional changes in response to stimuli.

[0070] In one feasible design, this application provides a method for training a neural network model, as follows: Acquire a source stimulus domain, which includes at least one stimulus data provided to each of a plurality of subjects, the stimulus data being used to stimulate the subjects’ senses. Acquire the source response domain, which includes the corresponding facial behavior data generated by each subject after being stimulated by at least one stimulus data. Obtain a classification label for each participant; the classification label is used to indicate whether the participant has a depressive mood disorder. A stimulus coding model is used to encode each stimulus data in the source stimulus domain to obtain the first vector representation of each stimulus data; A facial behavior coding model is used to encode each facial behavior data in the source response domain, resulting in a second vector representation of each facial behavior data. Using each first vector representation, each corresponding second vector representation, and the corresponding subject's classification label, a Transformer model, a stimulus encoding model, and a facial behavior encoding model are trained. This enables the Transformer model to learn the relationship features between each stimulus data and each corresponding facial behavior data, and to learn the common relationship features of the classification labels corresponding to subjects with depressive mood disorders from the relationship features of each data pair corresponding to subjects with depressive mood disorders. This enables the stimulus encoding model to learn to extract emotional features from the stimulus data, and enables the facial behavior encoding model to extract facial features of subjects with depressive mood disorders from the facial behavior data.

[0071] The content of the stimulus data and facial behavior data can be found in the above-mentioned target stimulus data and target facial behavior data, respectively, and will not be repeated here.

[0072] It should be understood that the classification labels in "obtaining the classification labels of each subject" refer to the actual classification labels.

[0073] For example, in order to ensure data quality, the data is preprocessed before encoding each stimulus data and / or before encoding each facial behavior data. The preprocessing method is the same as the method for preprocessing the target stimulus data described above, and will not be repeated here.

[0074] It should be understood that the loss function used to train the above model can be set according to actual needs, and this application does not impose any restrictions on it.

[0075] The above example uses stimulus-response data pairs (including stimulus data and corresponding facial behavior data) and corresponding subject classification labels to train a neural network model, a stimulus encoding model, and a facial behavior encoding model. This enables the stimulus encoding model to learn to extract emotional features from the stimulus data, and the facial behavior encoding model to extract facial features of subjects with depressive mood disorders from the facial behavior data. The neural network model can integrate information from the stimulus-response data pairs, learning more fundamental information for identifying depressive mood disorders by extracting common features from multiple stimulus-response data pairs of individuals with depressive mood disorders. Therefore, compared to current methods that only study the differences in facial behavior data between healthy and depressed individuals for depressive mood disorder identification, the neural network model trained with multiple stimulus-response data pairs in this application can identify depressive mood disorders through fundamental depressive mood disorder identification information. This means that regardless of the content and form of the target stimulus data, the neural network model can identify depressive mood disorders through the relationship features between the target stimulus data and facial behavior data, enhancing the generalization performance of the neural network model and having significant implications for deploying the model in real-world medical scenarios.

[0076] Other implementation methods and effects in the above neural network model training method embodiments can be found in the descriptions of methods S110 to S150, and will not be repeated here.

[0077] For example, the source stimulus domain includes positive type stimulus data, neutral type stimulus data, and negative type stimulus data. Positive type stimulus data expresses positive emotions, neutral type stimulus data has no emotional bias, and negative type stimulus data expresses negative emotions.

[0078] In the example above, training the model by increasing the variety of data included in the source stimulus domain can enhance the model's robustness.

[0079] In one feasible design, this application also provides a method for improving the performance of a neural network model, the method comprising: Acquire the target stimulus domain, which includes at least one stimulus data provided to each of the multiple subjects, and each stimulus data in the target stimulus domain is different from each stimulus data in the source stimulus domain. Obtain the target response domain, which includes the corresponding facial behavior data generated by each subject after passing through at least one stimulus data point in the target stimulus domain; Obtain the classification label for each participant; Encode each stimulus data in the target stimulus domain to obtain the first vector representation of each stimulus data in the target stimulus domain; Encode each facial behavior data in the target response domain to obtain a second vector representation of each facial behavior data in the target response domain; The trained Transformer model is evaluated using each first vector representation of the target stimulus domain, the corresponding second vector representation of the target response domain, and the classification labels of the subjects. The evaluation results are then obtained. Improve the Transformer model based on the evaluation results.

[0080] For example, the parameters and architecture of the neural network model can be modified based on the evaluation results. The neural network model can be adjusted according to actual needs; this application does not impose any limitations on this.

[0081] Similarly, the performance of the stimulus encoding model and the facial behavior encoding model can be evaluated simultaneously to obtain the corresponding evaluation results and improve the two models.

[0082] It should be understood that the target stimulus domain and the source stimulus domain do not include the same stimulus data. Since facial behavior data is the subject's response based on stimulus data, the target response domain and the source response domain naturally do not include the same response data.

[0083] To further enhance the generalization performance of the neural network model, this application uses a target stimulus domain that is completely different from the source stimulus domain to evaluate the performance of the neural network model. The improved neural network model has stronger generalization performance.

[0084] The method of training a neural network model using the source stimulus domain and source response domain, and then testing the performance of the neural network model on the target stimulus domain and target response domain, is similar to the concept of domain adaptation in transfer learning.

[0085] For example, the domain, ,in It is the feature space, where It is the distribution space of the response domain. It is the distribution space of the stimulus domain. It is a joint distribution. The task is defined as... , where Y is the classification label space. Prior distribution and likelihood function It is learned from the training dataset, and the data format is... .

[0086] The cross-domain (i.e., the data transfer problem from the source stimulus domain and source response domain to the target stimulus domain and target response domain) problem can be defined as learning a function To complete the task, you can... (The source domain, including the source stimulus domain and the source response domain, generalizes well to) (Target domain, including the target stimulus domain and the target response domain), where The tasks in both domains are the same. The goal of this problem is to leverage knowledge learned from the source domain to improve the model's performance in the target domain.

[0087] Based on this, and considering that it is practically impossible to exhaust all stimuli, additional constraints need to be imposed, namely in the source domain. No methods can be used during the training of the model. The information obtained is called the zero-shot cross-domain transfer learning task. The learned function... It is necessary to maximize in the target domain The predictive performance on [the target], the transfer learning process can be formalized as […]. The parameters of the fully trained function on the source domain (where the training set is located) are: function It is necessary to best utilize each The samples in the dataset are correctly classified, and the classification function is: N is the number of samples.

[0088] The model uses the cross-entropy loss function, and the evaluation metrics in the performance evaluation process include accuracy, recall, precision, and F1 score.

[0089] TP (True Positives): True Positives are those predicted to be positive and are indeed positive. FP (False Positives): False positives are those that are predicted to be positive but are actually negative. FN (false Negatives): False negatives, predicted as negative but actually positive; TN (True Negatives): True negatives are those that are predicted to be negative and are indeed negative.

[0090] The cross-entropy loss function is shown in the following formula (9): Formula (9); in, The label representing sample i, This represents the probability that sample i is predicted to be of the positive class, and N is the number of samples.

[0091] accuracy ; Recall rate ; Accuracy ; F1 score .

[0092] Other implementation methods and effects of the above-mentioned improved neural network model can be found in the descriptions of methods S110 to S150, and will not be repeated here.

[0093] In this embodiment, target stimulus data, which stimulates the senses of the target subject, and the subject's response information (i.e., target facial behavior data) are first acquired. The target stimulus data is encoded to obtain a first target vector representation, which transforms the target stimulus data into encoded data (i.e., a vector representation) with a certain structure and meaning, enabling the Transformer model to interpret and understand the target stimulus data. Similarly, the target facial behavior data is encoded into a second target vector representation, enabling the Transformer model to interpret and understand the target facial behavior data. The first and second target vector representations are input into the Transformer model, allowing the Transformer model to identify whether the target subject has a depressive mood disorder by recognizing the relationship characteristics between the target stimulus data and the target facial behavior data. Compared to current methods that only study the differences in facial behavior data between healthy and depressed individuals for depressive mood disorder identification, the Transformer model used in this application integrates information from the target stimulus data and the target facial behavior data, comprehensively understanding the subject's emotional changes from receiving the target stimulus data to responding based on the essential relationship characteristics between the two, thereby accurately identifying whether the subject has a depressive mood disorder. The depressive mood disorder identification method provided in this application can identify depressive mood disorders regardless of the content and form of the target stimulus data, thus enhancing the generalization performance of the Transformer model and having significant implications for deploying the model in real-world medical scenarios.

[0094] This application also provides a device for identifying depressive mood disorders based on facial behavioral data, such as... Figure 7 As shown, it includes: The data acquisition module is used to acquire target stimulus data provided to the target subjects, and the target stimulus data is used to stimulate the senses of the target subjects. The data acquisition module is used to acquire the target facial behavior data generated by the target subject after being stimulated by the target stimulus data; The stimulus encoding model is used to encode target stimulus data to obtain a first vector representation of the emotional features of the target stimulus data; A facial behavior encoding model is used to encode target facial behavior data to obtain a second vector representation of the facial features of the target facial behavior data; The depressive mood disorder identification module is used to input the first vector representation and the second vector representation into the Transformer model, and obtain the classification result output by the Transformer model by identifying the relationship features between the target stimulus data and the target facial behavior data. The classification result is used to indicate whether the target subject has a depressive mood disorder.

[0095] In one feasible design, the target facial behavior data is video data. The facial behavior coding model is implemented by encoding the target facial behavior data to obtain a second vector representation of the facial features of the target facial behavior data in the following way: A facial behavior encoding model is used to encode target facial behavior data using a three-dimensional convolutional neural network to obtain a first intermediate vector representation; The facial behavior coding model is also used to input the first intermediate vector representation into the long short-term memory network to obtain the second intermediate vector representation output by the long short-term memory network. The facial behavior coding model is also used to input the sum of the first intermediate vector representation and the second intermediate vector representation into the self-attention layer to obtain the third intermediate vector representation output by the self-attention layer. The facial behavior coding model is also used to input the sum of the second intermediate vector representation and the third intermediate vector representation into the decoder to obtain the second vector representation of the facial features of the target facial behavior data output by the decoder.

[0096] In one feasible design, the target stimulus data is video data. The stimulus encoding model encodes the target stimulus data in the following way to obtain a first vector representation of the emotional features of the target stimulus data: Input the target stimulus data into the three-dimensional residual network to obtain the first vector representation of the emotional features of the target stimulus data output by the three-dimensional residual network.

[0097] In one feasible design, the device also includes a model training module, wherein, The data acquisition module is also used to acquire the source stimulus domain, which includes at least one stimulus data provided to each of the multiple subjects, and the stimulus data is used to stimulate the subjects' senses. The data acquisition module is also used to acquire the source response domain, which includes the corresponding facial behavior data generated by each subject after being stimulated by at least one stimulus data. The data acquisition module is also used to obtain the classification label for each subject, which is used to indicate whether the subject has a depressive mood disorder. The stimulus coding model is also used to encode each stimulus data in the source stimulus domain to obtain a first vector representation of each stimulus data. The facial behavior coding model is also used to encode each facial behavior data in the source response domain to obtain a second vector representation of each facial behavior data. The model training module is used to train the Transformer model, the stimulus encoding model, and the facial behavior encoding model using each first vector representation, each corresponding second vector representation, and the corresponding subject's classification label. This enables the Transformer model to learn the relationship features between each stimulus data and each corresponding facial behavior data, and to learn the common relationship features of the classification labels corresponding to subjects with depressive mood disorders from the relationship features of each data pair corresponding to subjects with depressive mood disorders. This enables the stimulus encoding model to learn to extract emotional features from the stimulus data, and enables the facial behavior encoding model to extract facial features of subjects with depressive mood disorders from the facial behavior data.

[0098] In a feasible design, the source stimulus domain includes positive, neutral, and negative stimulus data. Positive stimulus data expresses positive emotions, neutral stimulus data has no emotional bias, and negative stimulus data expresses negative emotions.

[0099] In one feasible design, the device also includes an evaluation module, wherein, The data acquisition module is also used to acquire the target stimulus domain, which includes at least one stimulus data provided to each of the multiple subjects, and each stimulus data in the target stimulus domain is different from each stimulus data in the source stimulus domain. The data acquisition module is also used to acquire the target response domain, which includes the corresponding facial behavior data generated by each subject after being stimulated by at least one stimulus data in the target stimulus domain. The data acquisition module is also used to obtain the classification label for each subject; The stimulus encoding model is also used to encode each stimulus data in the target stimulus domain to obtain a first vector representation of each stimulus data in the target stimulus domain; The facial behavior coding model is also used to encode each facial behavior data in the target response domain to obtain a second vector representation of each facial behavior data in the target response domain; The evaluation module is used to evaluate the performance of the trained Transformer model using each first vector representation of the target stimulus domain, the corresponding second vector representation of the target response domain, and the classification labels of the corresponding subjects, and to obtain the evaluation results. The evaluation module is used to improve the Transformer model based on the evaluation results.

[0100] In one feasible design, the device further includes a preprocessing module for preprocessing the target stimulus data before encoding it; and / or, Used to preprocess target facial behavior data before encoding it.

[0101] In one feasible design, the decoder includes at least one first fully connected layer and at least one first activation layer.

[0102] In a feasible design, the three-dimensional residual network comprises multiple residual blocks.

[0103] Other implementation methods and effects of the above-mentioned device can be found in the description of the embodiment of the method for identifying depressive mood disorders based on facial behavior data, and will not be repeated here.

[0104] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0105] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0106] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0107] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0108] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0109] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for identifying depressive mood disorders based on facial behavioral data, characterized in that, include: Acquire target stimulus data provided to the target subject, the target stimulus data being used to stimulate the sensory organs of the target subject; Acquire the target facial behavior data generated by the target subject after being stimulated by the target stimulus data; The target stimulus data is encoded to obtain a first vector representation of the emotional features of the target stimulus data; Encoding the target facial behavior data to obtain a second vector representation of the facial features of the target facial behavior data; the type of the target facial behavior data is video data, and the encoding of the target facial behavior data to obtain the second vector representation of the facial features of the target facial behavior data includes: After encoding the target facial behavior data using a three-dimensional convolutional neural network, a first intermediate vector representation is obtained; Input the first intermediate vector representation into the Long Short-Term Memory Network to obtain the second intermediate vector representation output by the Long Short-Term Memory Network; The sum of the first intermediate vector representation and the second intermediate vector representation is input into the self-attention layer to obtain the third intermediate vector representation output by the self-attention layer; The second intermediate vector representation and the third intermediate vector representation are input into the decoder to obtain the second vector representation of the facial features of the target facial behavior data output by the decoder; Input the first vector representation and the second vector representation into the Transformer model to obtain the classification result output by the Transformer model by identifying the relationship features between the target stimulus data and the target facial behavior data. The classification result is used to indicate whether the target subject has a depressive mood disorder.

2. The method according to claim 1, characterized in that, The target stimulus data is video data. Encoding the target stimulus data to obtain a first vector representation of the emotional features of the target stimulus data includes: The target stimulus data is input into a three-dimensional residual network to obtain a first vector representation of the emotional features of the target stimulus data output by the three-dimensional residual network.

3. The method according to claim 1, characterized in that, The method further includes: Acquire a source stimulus domain, the source stimulus domain comprising at least one stimulus data provided to each of a plurality of subjects, the stimulus data being used to stimulate the subject’s senses. Obtain the source response domain, which includes the corresponding facial behavior data generated by each subject after being stimulated by at least one of the stimulus data; Obtain a classification label for each subject, the classification label being used to indicate whether the subject has a depressive mood disorder; Each stimulus data in the source stimulus domain is encoded using a stimulus coding model to obtain a first vector representation of each stimulus data. Each facial behavior data in the source response domain is encoded using a facial behavior coding model to obtain a second vector representation of each facial behavior data. Using each of the first vector representations, each corresponding second vector representation, and the classification label of the corresponding subject, the Transformer model, the stimulus encoding model, and the facial behavior encoding model are trained. This enables the Transformer model to learn the relationship features between each stimulus data and each corresponding facial behavior data, and to learn the common relationship features of the classification labels corresponding to depressive mood disorders from the relationship features of each data pair corresponding to subjects with depressive mood disorders. This enables the stimulus encoding model to learn to extract emotional features from the stimulus data, and enables the facial behavior encoding model to extract facial features of subjects with depressive mood disorders from the facial behavior data.

4. The method according to claim 3, characterized in that, The source stimulus domain includes positive, neutral, and negative stimulus data. The positive stimulus data expresses positive emotions, the neutral stimulus data has no emotional bias, and the negative stimulus data expresses negative emotions.

5. The method according to claim 3, characterized in that, The method further includes: Acquire a target stimulus domain, the target stimulus domain comprising at least one stimulus data provided to each of a plurality of subjects, each stimulus data in the target stimulus domain being different from each stimulus data in the source stimulus domain; Obtain the target response domain, which includes the corresponding facial behavior data generated by each subject after being stimulated by at least one of the stimulus data in the target stimulus domain; Obtain the classification label for each of the subjects; Encode each stimulus data in the target stimulus domain to obtain a first vector representation of each stimulus data in the target stimulus domain; Encode each facial behavior data in the target response domain to obtain a second vector representation of each facial behavior data in the target response domain; The trained Transformer model is evaluated using each of the first vector representations of the target stimulus domain, the corresponding second vector representations of the target response domain, and the classification labels of the subjects, and the evaluation results are obtained. The Transformer model is improved based on the evaluation results.

6. The method according to claim 1, characterized in that, Before encoding the target stimulus data, the method further includes: preprocessing the target stimulus data; and / or, Before encoding the target facial behavior data, the method further includes preprocessing the target facial behavior data.

7. The method according to claim 1, characterized in that, The decoder includes at least one first fully connected layer and at least one first activation layer.

8. The method according to claim 3, characterized in that, The three-dimensional residual network consists of multiple residual blocks.

9. A device for identifying depressive mood disorders based on facial behavioral data, characterized in that, include: The data acquisition module is used to acquire target stimulus data provided to the target subject, and the target stimulus data is used to stimulate the sensory organs of the target subject. The data acquisition module is used to acquire the target facial behavior data generated by the target subject after being stimulated by the target stimulus data; A stimulus encoding model is used to encode the target stimulus data to obtain a first vector representation of the emotional features of the target stimulus data; A facial behavior encoding model is used to encode the target facial behavior data to obtain a second vector representation of the facial features of the target facial behavior data; the type of the target facial behavior data is video data, and the encoding of the target facial behavior data to obtain the second vector representation of the facial features of the target facial behavior data includes: After encoding the target facial behavior data using a three-dimensional convolutional neural network, a first intermediate vector representation is obtained; Input the first intermediate vector representation into the Long Short-Term Memory Network to obtain the second intermediate vector representation output by the Long Short-Term Memory Network; The sum of the first intermediate vector representation and the second intermediate vector representation is input into the self-attention layer to obtain the third intermediate vector representation output by the self-attention layer; The second intermediate vector representation and the third intermediate vector representation are input into the decoder to obtain the second vector representation of the facial features of the target facial behavior data output by the decoder; The depressive mood disorder identification module is used to input the first vector representation and the second vector representation into the Transformer model to obtain the classification result output by the Transformer model by identifying the relationship features between the target stimulus data and the target facial behavior data. The classification result is used to indicate whether the target subject has a depressive mood disorder.

Images