Depression symptom feature acquisition device, system, method and storage medium

Abstract

The present application relates to a device, system, method, and storage medium for acquiring depressive symptom features based on facial expression changes. The device comprises: a video stream acquisition module for acquiring a video stream of a target user's facial expressions, wherein the video stream contains multiple facial expressions of the target user; a first feature tensor acquisition module for acquiring the number of video frames in the video stream and the facial expression features of each frame in the video stream, and obtaining a first feature tensor based on the number of video frames and the facial expression features of each frame; and a second feature vector acquisition module for inputting the first feature tensor into a pre-trained self-attention model to obtain a second feature vector output by the pre-trained self-attention model, wherein the second feature vector represents the depressive symptom features of the target user. The above device can improve the accuracy of the analysis results of depressive state features.

Description

Technical Field

[0001] The present application relates to the technical field of computer data processing, and in particular to a device, system, method and storage medium for acquiring depressive symptom characteristics based on facial expression changes. Background Art

[0002] Currently, the diagnosis of depression is primarily based on clinical questionnaires, psychologists' observations, and subjective descriptions. These methods are susceptible to physician bias and clinical experience, leading to misdiagnosis. Existing methods or systems for assisting in the diagnosis of depression, such as those for identifying depression through facial expressions, essentially treat all facial expressions in a recorded video as equivalent. Alternatively, each frame in a video stream is matched against a Facial Expression Coding System (FACS), and different features from the FACS are combined to distinguish different facial expressions.

[0003] When processing video streams of facial expressions, facial expression coding systems use recurrent neural networks for modeling. This significantly limits the length of the input video stream. Furthermore, due to the inherent characteristics of recurrent neural networks, earlier input data has a smaller impact on the results. However, changes in facial expressions in people with depression are related to psychological changes caused by their environment and stimuli, and are not entirely influenced by previous changes in expression. Therefore, recurrent neural networks have inherent shortcomings when processing video expressions.

[0004] When recording the facial expressions of users with depression, the video streams recorded by the users are long and contain many frames. Generally, the facial expressions between adjacent frames vary very little. When micro-expressions do occur, these changes are what are meaningful for assessing depressive mood. However, micro-expressions change rapidly and occur in relatively few frames. Using a recurrent neural network to downsample the original video stream may exclude valid frames. Therefore, using a traditional facial expression coding system to assist in analyzing the characteristics of depression in users with depression results in low accuracy. Furthermore, averaging adjacent frames of the video stream blurs the changes in the user's expressions. Therefore, using the traditional method of treating all facial expressions as equivalent to analyze the characteristics of depression in users with depression results in low accuracy. Summary of the Invention

[0005] Based on this, it is necessary to provide a device, system, method and storage medium for acquiring depressive symptom characteristics based on facial expression changes, which can improve the accuracy of depression state characteristic analysis results in response to the above technical problems.

[0006] A device for acquiring depressive symptom features based on facial expression changes includes: a video stream acquisition module, used to acquire a video stream of a target user's facial expressions, the video stream containing multiple facial expressions of the target user; a first feature tensor acquisition module, used to acquire the number of video frames in the video stream and the facial expression features of each frame of video in the video stream, and obtain a first feature tensor based on the number of video frames and the facial expression features of each frame of video; a second feature vector acquisition module, used to input the first feature tensor into a pre-trained self-attention model to obtain a second feature vector output by the pre-trained self-attention model, the second feature vector representing the depressive symptom features of the target user, and the pre-trained self-attention model is used to determine the importance of the facial expression features of each frame of video corresponding to the depressive state based on the first feature tensor and output a second feature vector based on the importance.

[0007] In one embodiment, a device for acquiring depressive symptom features based on facial expression changes also includes a self-attention model construction module, which includes: a third eigenvector acquisition unit, used to determine a third eigenvector according to S0=Q*K0, wherein S0 represents the third eigenvector, the third eigenvector is a vector representing the importance of each frame of video in the first eigentensor, Q represents the first eigentensor, and K0 is a representation vector representing the difference in facial expressions of corresponding video frames in the video stream in distinguishing between depressed users and healthy users; a position information acquisition unit, used to determine the position information of n values ​​of S0 ranked in the top n places according to Pos=argTopn(S0), Pos represents the position information of n values ​​of S0 ranked in the top n places, and Topn represents the position information of the median size of S0 extracted. The first n values, arg represents the position information of the extracted n values, and n represents a positive integer; a fourth eigenvector acquisition unit is used to determine the fourth eigenvector according to Q1=Q[Pos,:], Q1 represents the fourth eigenvector, Q[Pos,:] represents the eigenvector obtained by extracting the elements of Pos from Q corresponding to the elements of each row in Q and combining the elements of each row; an importance acquisition unit is used to determine the importance of each element in the fourth eigenvector according to S1=softmax(Q1*K1), S1 represents the importance of each element in the fourth eigenvector, K1 represents the representation vector of the facial expression difference of the corresponding video frame in Q1 in distinguishing between depressed users and healthy users, and softmax represents the softmax function; a depressive symptom feature acquisition unit is used to determine the importance of each element in the fourth eigenvector according to P=S1 T *Q1 determines the depressive symptom characteristics, and P represents the second eigenvector.

[0008] In one embodiment, a device for acquiring depressive symptom features based on facial expression changes also includes a contribution vector acquisition module, which is used to obtain a feature vector of the importance of each frame of video in a video stream output by the pre-trained self-attention model relative to a second feature vector when a first feature tensor is input into a pre-trained self-attention model; wherein the self-attention model obtains the feature vector of the importance of each frame of video relative to the second feature vector according to C=softmax(S0), and C represents the feature vector of the importance of each frame of video relative to the second feature vector.

[0009] In one embodiment, a device for acquiring depressive symptom features based on facial expression changes also includes a training module of a self-attention model. The training module of the self-attention model includes: a training data acquisition unit, used to acquire video data of facial expressions of users with depression and video data of facial expressions of healthy users; a random value setting unit, used to randomly initialize K0 and K1; a model training unit, used to input the video data of facial expressions of users with depression and video data of facial expressions of healthy users into the self-attention model, and update K0 and K1 by calculating the cross-entropy loss and using the gradient update algorithm of deep learning and the back-propagation algorithm to obtain a pre-trained self-attention model.

[0010] In one embodiment, the first feature tensor acquisition module includes: a facial expression feature acquisition unit, used to input the video stream into a pre-trained convolutional neural network to obtain a feature vector of the facial expression features of each frame of video in the video stream output by the pre-trained convolutional neural network; a first feature tensor acquisition unit, used to splice the feature vectors of the facial expression features of each frame of video in the video stream side by side according to the number of video frames to obtain a first feature tensor.

[0011] In one embodiment, a device for acquiring depressive symptom features based on facial expression changes also includes a convolutional neural network training module, and the convolutional neural network training module includes: a first training unit, used to acquire a picture set, input the picture set into the convolutional neural network, and obtain a preliminary convolutional neural network, which is a network model that can represent the abstract content of the picture; a second training unit, used to acquire various types of facial expression pictures, input various types of facial expression pictures into the preliminary convolutional neural network, and obtain a pre-trained convolutional neural network.

[0012] In one embodiment, a device for acquiring depressive symptom features based on facial expression changes also includes a depressive expression analysis module, which is used to input the second feature vector into a pre-trained multi-layer perceptron to obtain an evaluation result of the depressive expression output by the pre-trained multi-layer perceptron; wherein the pre-trained multi-layer perceptron is trained using facial expression data of users with depression and facial expression data of healthy users.

[0013] A system for acquiring depressive symptom features based on facial expression changes includes a server and a terminal; the terminal is used to collect a video stream of a target user's facial expressions and send the video stream to the server, wherein the video stream contains multiple facial expressions of the target user; the server is used to acquire the video stream sent by the terminal, and obtain the number of video frames in the video stream and the facial expression features of each frame in the video stream, and obtain a first feature tensor based on the number of video frames and the facial expression features of each frame, and input the first feature tensor into a pre-trained self-attention model to obtain a second feature vector output by the pre-trained self-attention model, wherein the second feature vector represents the depressive symptom features of the target user, and the pre-trained self-attention model is used to determine the importance of the facial expression features of each frame of video corresponding to the depressive state based on the first feature tensor and output a second feature vector based on the importance.

[0014] A method for acquiring depressive symptom features based on facial expression changes includes: acquiring a video stream of a target user's facial expressions, the video stream containing multiple facial expressions of the target user; acquiring the number of video frames in the video stream and the facial expression features of each frame in the video stream, and obtaining a first feature tensor based on the number of video frames and the facial expression features of each frame; inputting the first feature tensor into a pre-trained self-attention model to obtain a second feature vector output by the pre-trained self-attention model, the second feature vector representing the depressive symptom features of the target user, and the pre-trained self-attention model being used to determine the importance of the facial expression features of each frame of video corresponding to the depressive state based on the first feature tensor and output a second feature vector based on the importance.

[0015] A computer-readable storage medium stores a computer program, which implements the steps of any of the above-mentioned method embodiments when executed by a processor.

[0016] The above-described apparatus, system, method, and storage medium for acquiring depressive symptom features based on facial expression changes acquire a video stream of a target user's facial expressions, the video stream containing multiple facial expressions of the target user. The apparatus then acquires the number of video frames in the video stream and the facial expression features of each frame in the video stream. Based on the number of video frames and the facial expression features of each frame, a first feature tensor is obtained. Therefore, the first feature tensor contains the facial expression features of each frame in the video stream, meaning that the first feature tensor can characterize the multiple facial expressions of the target user. Furthermore, the first feature tensor is input into a pre-trained self-attention model to obtain a second feature vector output by the pre-trained self-attention model, which characterizes the depressive symptom features of the target user. The pre-trained self-attention model is configured to determine the importance of the facial expression features of each frame in the video stream relative to the depressive state based on the first feature tensor and output a second feature vector based on the importance. Consequently, the importance of the facial expression features of each frame in the video stream relative to the depressive state is analyzed, and a second feature vector representing the depressive symptom features of the target user is output based on the importance, thereby improving the accuracy of the depressive symptom features of the target user. In addition, the self-attention model has no restrictions on the length of the video stream, so the video stream of the target user's facial expressions recorded does not need to be deleted. Analysis can be performed based on all the changes in the target user's facial expressions, thereby improving the accuracy of the target user's depressive symptom characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a structural block diagram of a device for acquiring depressive symptom features based on facial expression changes in one embodiment;

[0018] Figure 2 A structural block diagram of a self-attention model building module in one embodiment;

[0019] Figure 3 1 is a structural block diagram of a training module of a self-attention model in one embodiment;

[0020] Figure 4 is a structural block diagram of a first feature tensor acquisition module in one embodiment;

[0021] Figure 5 is a structural block diagram of a training module of a convolutional neural network in one embodiment;

[0022] Figure 6 A schematic diagram of a framework of a system for acquiring depressive symptom features based on facial expression changes in one embodiment;

[0023] Figure 7 A schematic diagram of a framework of a depression expression analysis model constructed by the server 602 in one embodiment;

[0024] Figure 8 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0026] This application provides a device for acquiring depressive symptom features based on facial expression changes, which is applied to a server. Figure 1 As shown, in one embodiment, a device 100 for acquiring depressive symptom features based on facial expression changes includes:

[0027] The video stream acquisition module 102 is configured to acquire a video stream of facial expressions of a target user, where the video stream includes multiple changing facial expressions of the target user.

[0028] In this embodiment, the target user refers to the user whose depressive symptom characteristics are to be obtained. In some cases, others or the target user himself may want to use the depressive symptom characteristics to understand whether the target user suffers from depression or has a tendency to depression. The terminal uploads a video stream of the target user's facial expressions, and the video stream contains the target user's changing facial expressions. Specifically, the terminal records the target user's facial expressions to obtain a video stream, and each frame of the video stream contains the target user's changing facial expressions. Therefore, when the server obtains the video stream of the target user's facial expressions, by analyzing the facial expressions of each frame of the video stream, the target user's changing micro-expression information can be obtained.

[0029] The first feature tensor acquisition module 104 is used to obtain the number of video frames in the video stream and the facial expression features of each frame in the video stream, and obtain a first feature tensor according to the number of video frames and the facial expression features of each frame.

[0030] In this embodiment, the first feature tensor acquisition module 104 determines the video length of the video stream and obtains the number of video frames. At the same time, the first feature tensor acquisition module 104 extracts the facial expression features of each frame of video. The first feature tensor acquisition module 104 can use the existing deep learning model to obtain facial expression features through transfer learning. Facial expression features represent the facial expression information of the user in the picture of the frame video, such as eyes, muscle changes, and light and dark contrast of the face caused by muscle changes. Further, the first feature tensor acquisition module 104 determines the first feature tensor based on the number of video frames and the facial expression features of each frame of video. For example, L is the video length of the video stream of facial expression, that is, the number of video frames of the video stream is L. M is the length of the facial expression features extracted from the expression pictures of each frame of video. Therefore, the first feature tensor acquisition module 104 can obtain L representation vectors of length M, and splice them in parallel to obtain the first feature tensor.

[0031] The second feature vector acquisition module 106 is used to input the first feature tensor into a pre-trained self-attention model to obtain a second feature vector output by the pre-trained self-attention model, where the second feature vector represents the depressive symptom characteristics of the target user. The pre-trained self-attention model is used to determine the importance of the facial expression characteristics of each frame of the video corresponding to the depressive state based on the first feature tensor and output the second feature vector based on the importance.

[0032] In this embodiment, a pre-trained self-attention model is stored in the second feature vector acquisition module 106. The self-attention model is obtained by model training using video data of facial expressions of users with depression and video data of facial expressions of healthy users. That is, the self-attention model is trained using feature vectors of facial expression features in the video data of facial expressions of users with depression and feature vectors of facial expression features in the video data of facial expressions of healthy users, and the model parameters are adjusted to obtain the pre-trained self-attention model. Therefore, the pre-trained self-attention model can determine the importance of the facial expression features of each frame of video corresponding to the depressive state based on the first feature tensor and output a second feature vector representing the depressive symptom characteristics of the target user based on the importance.

[0033] The above-mentioned depressive symptom feature acquisition device based on facial expression changes obtains a video stream of a target user's facial expressions, the video stream containing multiple facial expressions of the target user. The device then obtains the number of video frames in the video stream and the facial expression features of each frame in the video stream. Based on the number of video frames and the facial expression features of each frame, a first feature tensor is obtained. Therefore, the first feature tensor contains the facial expression features of each frame in the video stream, meaning that the first feature tensor can characterize the multiple facial expressions of the target user. Furthermore, the first feature tensor is input into a pre-trained self-attention model to obtain a second feature vector output by the pre-trained self-attention model, which characterizes the depressive symptom features of the target user. The pre-trained self-attention model is configured to determine the importance of the facial expression features of each frame in the video stream relative to the depressive state based on the first feature tensor and output a second feature vector based on the importance. Therefore, the device can analyze the importance of the facial expression features of each frame in the video stream relative to the depressive state and output a second feature vector based on the importance, thereby improving the accuracy of the depressive symptom features of the target user. In addition, the self-attention model has no restrictions on the length of the video stream, so the video stream of the target user's facial expressions recorded does not need to be deleted. Analysis can be performed based on all the changes in the target user's facial expressions, thereby improving the accuracy of the target user's depressive symptom characteristics.

[0034] In one embodiment, the apparatus for acquiring depressive symptom features based on facial expression changes further includes a self-attention model building module. The self-attention model building module is used to build a self-attention model. Figure 2 As shown in Figure 2, the self-attention model building blocks include:

[0035] The third eigenvector acquisition unit 202 is used to determine the third eigenvector according to S0=Q*K0, wherein S0 represents the third eigenvector, the third eigenvector is a vector representing the importance of each frame of video in the first eigentensor, Q represents the first eigentensor, and K0 is a representation vector representing the difference in facial expressions of corresponding video frames in the video stream in distinguishing between depressed users and healthy users.

[0036] The position information acquisition unit 204 is used to determine the position information of the n values ​​of S0 that are ranked in the top n places in terms of median size according to Pos=argTopn(S0), where Pos represents the position information of the n values ​​of S0 that are ranked in the top n places in terms of median size, Topn represents extracting the n values ​​of S0 that are ranked in the top n places in terms of median size, arg represents the position information of the extracted n values, and n represents a positive integer.

[0037] The fourth eigenvector acquisition unit 206 is used to determine the fourth eigenvector according to Q1=Q[Pos,:], where Q1 represents the fourth eigenvector, and Q[Pos,:] represents the eigenvector obtained by extracting the elements of Pos from Q and corresponding to the elements of each row in Q and combining the elements of each row.

[0038] The importance acquisition unit 208 is used to determine the importance of each element in the fourth eigenvector according to S1=softmax(Q1*K1), where S1 represents the importance of each element in the fourth eigenvector, K1 represents the representation vector of the facial expression difference of the corresponding video frame in Q1 in distinguishing between depressed users and healthy users, and softmax represents the softmax function.

[0039] The depressive symptom feature acquisition unit 210 is configured to obtain the depressive symptom feature according to P=S1. T *Q1 determines the depressive symptom characteristics, and P represents the second eigenvector.

[0040] In this embodiment, the calculation method of the self-attention model is as follows:

[0041] i) S0 = Q*K0;

[0042] ii) Pos = argTopn(S0), where Pos is the position information of the n values ​​in S0 that are ranked first in descending order, Topn represents extracting the n values ​​in S0 that are ranked first in order, and arg represents extracting the position information of these n values;

[0043] iii) Q1 = Q[Pos,:], S1 = softmax(Q1*K1), where Q[Pos,:] represents the vector obtained by extracting and combining the elements of each row in Q corresponding to each element in Pos;

[0044] iv) P=S1 T *Q1;

[0045] v) Return P.

[0046] In the above self-attention model, among S0, Q, K0, Q1, K1, S1, and P, Q and Q1 are tensors, and S0, K0, K1, S1, and P are vectors. The initial values ​​of K0 and K1 are random values, which are adjusted by the back-propagation algorithm of deep learning during the model training process, and finally the parameter values ​​suitable for this embodiment are obtained. Q is a tensor with L rows and M columns, where L is the video length of the input facial expression video stream, and M is the length of the facial expression feature extracted from the expression picture of each frame of the video stream. The facial expression feature can be obtained by transfer learning using an existing deep learning model. The facial expression feature is an overall representation of the facial expression of the target user in a frame of video. For example, the facial expression feature is a feature vector of a numerical value obtained based on information such as changes in the eyes, muscles, and light and dark contrast caused by muscle changes on the face. K0 is a vector with M rows and 1 column. K0 represents the difference in facial expressions between the corresponding video frame in the video stream and healthy users. It is used to calculate the importance of each video frame in the video stream and extract the n most important frames. The calculation method is the same as in i) and ii) of the self-attention model above. Q1 is a tensor with n rows and M columns, which contains the features of the n most contributing frames in Q. K1 is a vector with M rows and 1 column. K1 represents the difference in facial expressions between the corresponding video frame in Q1 and healthy users. It is used to calculate the relative importance of each frame in Q1. It is also normalized using the softmax function to obtain S1. S1 is used to perform a weighted sum of each frame in Q1 to obtain the final overall feature for depressive expression assessment. In the above self-attention model, the returned result P is an abstract vector of the facial expressions and their changes in the key frames in the input video stream.

[0047] The self-attention model constructed in this embodiment can effectively calculate the degree to which the facial expression features of each frame in a video stream correspond to a depressive state, and perform key frame selection to identify the target user's depressive symptom characteristics. During the extraction process of the target user's depressive symptom characteristics, the contribution of the facial expression features in each video frame to the depressive state is considered, thereby improving the accuracy of the target user's depressive symptom characteristics. Furthermore, the constructed self-attention model has no restrictions on the length of the input video stream, and can obtain the target user's depressive symptom characteristics in a single calculation, reducing the amount of computation and thus improving the efficiency of video data processing.

[0048] In one embodiment, the apparatus for acquiring depressive symptom features based on facial expression changes further includes a contribution vector acquisition module. The contribution vector acquisition module is configured to, when the first feature tensor is input into a pre-trained self-attention model, obtain a feature vector representing the importance of each frame of video in a video stream output by the pre-trained self-attention model relative to a second feature vector. The self-attention model obtains the feature vector representing the importance of each frame of video relative to the second feature vector based on C = softmax(S0), where C represents the feature vector representing the importance of each frame of video relative to the second feature vector.

[0049] In this embodiment, the contribution vector acquisition module is used to calculate the importance of each frame of video in the video stream relative to the final result output by the self-attention model. In the above embodiment, the calculation method of the self-attention model also includes C=softmax(S0), and the output result of the self-attention model also includes C. Among them, C is an evaluation of the importance of each frame of video in the input video stream relative to the final model output result. Each position in C corresponds to each frame of video in the input video stream, the value of each position in C is between 0-1, and the sum of all elements in C is 1. Therefore, through the output C of the self-attention model, we can intuitively understand the processing results of the self-attention model on each frame of video in the video stream, which is convenient for the research and development of the management of the self-attention model.

[0050] In one embodiment, the apparatus for acquiring depressive symptom features based on facial expression changes further includes a training module for a self-attention model. Figure 3 As shown in Figure 2, the training modules of the self-attention model include:

[0051] The training data acquisition unit 302 is configured to acquire video data of facial expressions of users with depression and video data of facial expressions of healthy users.

[0052] The random value setting unit 304 is used to randomly initialize K0 and K1.

[0053] The model training unit 306 is used to input the video data of facial expressions of users with depression and the video data of facial expressions of healthy users into the self-attention model, calculate the cross-entropy loss and use the gradient update algorithm of deep learning and the backpropagation algorithm to update K0 and K1, thereby obtaining a pre-trained self-attention model.

[0054] In this embodiment, referring to the calculation method of the self-attention model described in the above embodiment, it can be seen that in the constructed self-attention model, the parameters for calculating the importance of each frame of the video frame, namely K0 and K1, are mainly constructed. Therefore, when the self-attention model is trained, the training data is used to train the self-attention model to determine the optimal values ​​of K0 and K1, thereby obtaining a pre-trained self-attention model. The main steps of the self-attention model training module include:

[0055] ① Initial stage: Since the difference in facial expressions between unknown depressed users and healthy users is maximized, K0 and K1 need to be randomly initialized, and the values ​​of each element in K0 and K1 are random values.

[0056] ② Training Phase: Video data of facial expressions from users with depression and healthy users are fed into the self-attention model. The model parameters of the self-attention model are updated by calculating the cross-entropy loss and applying the gradient update and backpropagation algorithms from deep learning. During this process, the values ​​of K0 and K1 are also updated. The entire model training process is considered the training process of the current deep learning model. After model training, K0 and K1 are optimized, initially unable to fully determine the contribution of each video frame to the model output, to a state where they can accurately determine the video frame that contributes most to the model output. The optimized K0 and K1 are the vectors that most closely represent the features of the video frames from the healthy and depressed users. In the input video stream, the greater the similarity between any feature vector representing the facial expression of each video frame and K0 or K1, the greater the contribution of that video frame to the model output.

[0057] Therefore, by training the self-attention model, a pre-trained self-attention model is obtained for extracting the depressive symptom characteristics of the target user in the video stream. Therefore, when the pre-trained self-attention model is used to process the video stream of the target user's facial expression, the obtained depressive symptom characteristics are more accurate, providing more accurate data for the assessment of the depression state.

[0058] In one embodiment, Figure 4 As shown, the first feature tensor acquisition module includes:

[0059] The facial expression feature acquisition unit 402 is used to input the video stream into a pre-trained convolutional neural network to obtain a feature vector of the facial expression feature of each frame of video in the video stream output by the pre-trained convolutional neural network.

[0060] The first feature tensor acquisition unit 404 is configured to concatenate the feature vectors of the facial expression features of each frame of video in the video stream side by side according to the number of video frames to obtain a first feature tensor.

[0061] In this embodiment, a video stream of facial expressions is input into a pre-trained convolutional neural network. The pre-trained convolutional neural network processes the frame images of each video frame in the video stream to obtain a feature vector for the facial expression features of each frame. For example, the pre-trained convolutional neural network extracts characteristic information such as facial features, their combinations, and the light and dark changes caused by muscle changes due to different expressions from each frame image, thereby obtaining a feature vector of length M. Furthermore, the first feature tensor acquisition unit 404 concatenates the feature vectors of the facial expression features of each video frame in the video stream side by side based on the number of video frames to obtain a first feature tensor. For example, each video frame corresponds to a feature vector of length M. Assuming that the input video stream has L frames, L feature vectors of length M are obtained. These feature vectors are concatenated side by side to obtain a first feature tensor with L rows and M columns. Therefore, the efficiency of facial expression feature extraction and the accuracy of facial expression features can be improved.

[0062] In one embodiment, the apparatus for acquiring depressive symptom features based on facial expression changes further includes a convolutional neural network training module, which is used to train the convolutional neural network to obtain a pre-trained convolutional neural network. Figure 5 As shown in Figure 2, the training modules of the convolutional neural network include:

[0063] The first training unit 502 is used to obtain a picture set and input the picture set into a convolutional neural network to obtain a preliminary convolutional neural network. The preliminary convolutional neural network is a network model that can represent the abstract content of the picture.

[0064] The second training unit 504 is used to obtain various types of facial expression pictures, and input the various types of facial expression pictures into the preliminary convolutional neural network to obtain a pre-trained convolutional neural network.

[0065] In this embodiment, the convolutional neural network is trained twice. The first training step may be as follows: the first training unit 502 obtains a picture set containing multiple image data. The first training unit 502 uses the picture set as training data for the convolutional neural network and performs model training on the convolutional neural network to obtain a preliminary convolutional neural network. The preliminary convolutional neural network is a network model capable of representing the abstract content of the image. Using the preliminary convolutional neural network, abstract features from the image can be extracted. The second training step may be as follows: the second training unit 504 uses images of various types of facial expressions as training data for the preliminary convolutional neural network and performs model training on the preliminary convolutional neural network to obtain a pre-trained convolutional neural network. For example, the second training step uses images of facial expressions of various emotions to perform classification training on the preliminary convolutional neural network. In other words, the preliminary convolutional neural network model learned for distinguishing other categories of images is transferred to distinguishing facial expression features, so that the image features learned by the preliminary convolutional neural network can be better adapted to facial data and can more accurately and preferentially extract abstract features of facial expressions. The pre-trained convolutional neural network can output a second eigenvector that characterizes the target user's depressive symptoms. This second eigenvector can be considered an abstract representation of information such as facial features, their combinations, and the light and dark variations caused by muscle changes associated with different expressions. Therefore, by training the convolutional neural network model using the convolutional neural network training module, a pre-trained convolutional neural network is obtained. This can improve the accuracy of facial expression feature extraction using the pre-trained convolutional neural network.

[0066] In one embodiment, the device for acquiring depressive symptom features based on facial expression changes also includes a depressive expression analysis module, which is used to input the second feature vector into a pre-trained multi-layer perceptron to obtain an evaluation result of the depressive expression output by the pre-trained multi-layer perceptron; wherein the pre-trained multi-layer perceptron is trained using facial expression data of users with depression and facial expression data of healthy users.

[0067] In this embodiment, the pre-trained multi-layer perceptron is trained using facial expression data of users with depression and facial expression data of healthy users, so the pre-trained multi-layer perceptron can distinguish the facial expressions of healthy users and users with depression. The second eigenvector characterizes the depressive symptom characteristics of the target user's facial expression. When the second eigenvector is input into the pre-trained multi-layer perceptron, the pre-trained multi-layer perceptron can process the second eigenvector and output an evaluation result of the depressive expression. The evaluation result of the depressive expression can be that the target user is a user with depression, or the target user is a healthy user, or the probability that the target user is a user with depression, or the probability that the target user is a healthy user. Therefore, an auxiliary depression assessment based on changes in the user's facial expression is achieved.

[0068] Each module in the above-mentioned apparatus for acquiring depressive symptom characteristics based on facial expression changes can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0069] In one embodiment, the present application also provides a system for acquiring depressive symptom features based on facial expression changes. Figure 6 As shown, the system for acquiring depressive symptom features based on facial expression changes includes a server 602 and a terminal 604. The terminal 604 is used to collect a video stream of the facial expressions of the target user and send the video stream to the server 602, wherein the video stream contains multiple facial expressions of the target user. The server 602 is used to acquire the video stream sent by the terminal 604, and obtain the number of video frames of the video stream and the facial expression features of each frame of the video stream, and obtain a first feature tensor based on the number of video frames and the facial expression features of each frame of the video, and input the first feature tensor into a pre-trained self-attention model to obtain a second feature vector output by the pre-trained self-attention model, the second feature vector represents the depressive symptom features of the target user, and the pre-trained self-attention model is used to determine the importance of the facial expression features of each frame of the video corresponding to the depressive state based on the first feature tensor and output a second feature vector based on the importance.

[0070] In this embodiment, the system for acquiring depressive symptom characteristics based on facial expression changes includes a server 602 and a terminal 604. Terminal 604 is installed with mobile software responsible for administering stimulation and recording a video stream of the target user's facial expression changes during the stimulation process. Server 602 serves as the cloud-side for the mobile software. The cloud-side processes and analyzes the recorded video stream of the target user and returns the depressive expression assessment results to the mobile software.

[0071] Server 602 is configured to process the target user's video stream to obtain the target user's depressive symptom characteristics in the video stream. The process of server 602 processing the target user's video stream is described in the description of the apparatus for obtaining depressive symptom characteristics based on facial expression changes in each of the above embodiments and will not be described in detail here.

[0072] In addition, server 602 is further configured to detect the target user's facial expression video stream. When the video stream is detected to meet the set requirements, the server 602 proceeds to the step of obtaining the number of video frames in the video stream and the facial expression characteristics of each frame in the video stream. When the video stream is detected to meet the set requirements, the server 602 sends feedback information to terminal 604. The feedback information indicates that the video stream does not meet the set requirements of server 602.

[0073] Specifically, server 602 extracts a frame image from each frame of the facial expression video stream transmitted by terminal 604 and uses facial recognition software to detect whether a face is present and whether the complete face is recorded. If a significant portion of the face is missing, such as 30%, feedback is sent to terminal 604, indicating that the face recording is incomplete and depressive expression analysis cannot be performed. Alternatively, facial recognition software is used to identify the face in the video stream and calculate the size of the area containing the face. If the size of the face changes significantly, feedback is sent to terminal 604, indicating that the video capture of the facial expression failed and that the target user moved significantly during the recording, making depressive expression analysis impossible.

[0074] In one implementation, the server 602 constructs a depression expression analysis model. Figure 7 As shown in the figure, the depression expression analysis model consists of three parts. The first part is a pre-trained convolutional neural network. The second part is AttBlock, which is a pre-trained self-attention model. The first part is MLP, which is a pre-trained multi-layer perceptron.

[0075] Regarding the first part: Since there is little data for depression expression analysis of depression, and the amount of data required for training to extract each frame of facial expression picture is very large. Therefore, a currently public, pre-trained deep learning model is used. Alternatively, the deep learning model for extracting facial expression features is pre-trained through an existing image classification dataset, so that the deep learning model can learn the abstract content of facial expression pictures. Furthermore, the deep learning model is trained through a facial expression classification dataset, so that the deep learning model can extract picture features from the abstract content of facial expression pictures, making the deep learning model more suitable for the extraction of facial expression features. Among them, the deep learning model is a convolutional neural network. Finally, the learned deep learning model is combined into Figure 7 In the overall model framework shown in Figure 1, we further improve the accuracy of extracting facial expression features in each frame of video through transfer learning methods.

[0076] In this implementation, a pre-trained resnet152 model is first used, and after removing the last classification layer of the resnet152 model, the remaining part of the resnet152 model is used as the output. Alternatively, other pre-trained image classification or recognition models are used, and the last several layers of the original model are removed, and the remaining part of the model is used as the output, and the output feature vector is used as the feature representation of the input image. Then, the pre-trained model is subjected to transfer learning to distinguish different facial expressions so that the learned image features can better represent the facial expression features of the human face. Each frame of the video stream of the target user's facial expression is input into the above-mentioned transfer-learned model to obtain an abstract feature vector of the facial expression of each frame of the video, and then the extracted feature vectors of each frame of the video are spliced ​​side by side into a vector to obtain the above-mentioned first feature tensor.

[0077] The first feature tensor is input into the AttBlock of the second part to obtain the feature vector corresponding to the video stream of the recorded facial expression for distinguishing depressive expressions, which is the second feature vector mentioned above.

[0078] The second eigenvector is input into the multilayer perceptron in the third part. This multilayer perceptron is a pre-trained model. The multilayer perceptron is trained using facial expression data from healthy and depressed users as a supervisory standard. The trained model becomes the multilayer perceptron in the third part.

[0079] In the overall model framework, the contribution of each frame of video in the input video stream to the evaluation result of depressive expression can be obtained from the AttBlock in the second part, that is, the value of C in the above embodiment is returned, and each bit of data in C corresponds to each frame of video in the video stream.

[0080] In this way, the server 602 inputs the video stream of the facial expression recorded by the mobile terminal software into the above-mentioned overall model framework, obtains the final evaluation result of the depressive expression, and sends the evaluation result of the depressive expression to the user of the terminal 604.

[0081] For more specific limitations on the server 602 of the system for acquiring depressive symptom features based on facial expression changes, please refer to the above limitations on the device for acquiring depressive symptom features based on facial expression changes, which will not be repeated here.

[0082] This application also provides a method for acquiring depressive symptom features based on facial expression changes, which is applied to Figure 6Taking the server 602 in the example as an example, the method includes the following steps: obtaining a video stream of facial expressions of a target user, wherein the video stream contains multiple facial expressions of the target user; obtaining the number of video frames in the video stream and the facial expression features of each frame in the video stream, and obtaining a first feature tensor according to the number of video frames and the facial expression features of each frame; inputting the first feature tensor into a pre-trained self-attention model to obtain a second feature vector output by the pre-trained self-attention model, wherein the second feature vector represents the depressive symptom features of the target user, and the pre-trained self-attention model is used to determine the importance of the facial expression features of each frame of video corresponding to the depressive state according to the first feature tensor and output a second feature vector according to the importance.

[0083] In one embodiment, the method for acquiring depressive symptom features based on facial expression changes also includes a self-attention model construction step, which includes: determining a third eigenvector according to S0=Q*K0, wherein S0 represents the third eigenvector, the third eigenvector is a vector representing the importance of each frame of video in the first eigentensor, Q represents the first eigentensor, and K0 is a representation vector representing the difference in facial expressions of corresponding video frames in the video stream in distinguishing between depression users and healthy users; determining the position information of the n values ​​of S0 ranked in the top n places according to Pos=argTopn(S0), Pos represents the position information of the n values ​​of S0 ranked in the top n places, and Topn represents the position information of the n values ​​of S0 extracted from S0 The n values ​​ranked in the top n positions by median size, arg represents the position information of the extracted n values, and n represents a positive integer; the fourth eigenvector is determined according to Q1=Q[Pos,:], Q1 represents the fourth eigenvector, Q[Pos,:] represents the eigenvector obtained by extracting the elements of Pos from Q and corresponding to the elements of each row in Q and combining the elements of each row; the importance of each element in the fourth eigenvector is determined according to S1=softmax(Q1*K1), S1 represents the importance of each element in the fourth eigenvector, K1 represents the representation vector of the facial expression difference of the corresponding video frame in Q1 in distinguishing between depression users and healthy users, and softmax represents the softmax function; according to P=S1 T *Q1 determines the depressive symptom characteristics, and P represents the second eigenvector.

[0084] In one embodiment, the method for acquiring depressive symptom features based on facial expression changes also includes: when the first feature tensor is input into a pre-trained self-attention model, a feature vector of the importance of each frame of video in the video stream output by the pre-trained self-attention model relative to the second feature vector is obtained; wherein the self-attention model obtains the feature vector of the importance of each frame of video relative to the second feature vector according to C=softmax(S0), and C represents the feature vector of the importance of each frame of video relative to the second feature vector.

[0085] In one embodiment, the method for acquiring depressive symptom features based on facial expression changes also includes a self-attention model training step, which includes: acquiring video data of facial expressions of users with depression and video data of facial expressions of healthy users; randomly initializing K0 and K1; inputting the video data of facial expressions of users with depression and video data of facial expressions of healthy users into the self-attention model, and updating K0 and K1 by calculating the cross-entropy loss and using the gradient update algorithm of deep learning and the back-propagation algorithm to obtain a pre-trained self-attention model.

[0086] In one embodiment, the above-mentioned steps of obtaining the number of video frames in the video stream and the facial expression features of each frame of video in the video stream and obtaining a first feature tensor based on the number of video frames and the facial expression features of each frame of video include: inputting the video stream into a pre-trained convolutional neural network to obtain a feature vector of the facial expression features of each frame of video in the video stream output by the pre-trained convolutional neural network; and splicing the feature vectors of the facial expression features of each frame of video in the video stream side by side according to the number of video frames to obtain a first feature tensor.

[0087] In one embodiment, the method for acquiring depressive symptom features based on facial expression changes also includes a convolutional neural network training step, which includes: acquiring a picture set, inputting the picture set into the convolutional neural network to obtain a preliminary convolutional neural network, and the preliminary convolutional neural network is a network model that can represent the abstract content of the picture; acquiring pictures of various types of facial expressions, and inputting the pictures of various types of facial expression into the preliminary convolutional neural network to obtain a pre-trained convolutional neural network.

[0088] In one embodiment, the method for acquiring depressive symptom features based on facial expression changes also includes a depressive expression analysis step, which includes: inputting the second eigenvector into a pre-trained multi-layer perceptron to obtain an evaluation result of the depressive expression output by the pre-trained multi-layer perceptron; wherein the pre-trained multi-layer perceptron is trained using facial expression data of users with depression and facial expression data of healthy users.

[0089] For more limitations on the technical features of the method for acquiring depressive symptom features based on facial expression changes, please refer to the above description of the corresponding modules or units in the device for acquiring depressive symptom features based on facial expression changes, which will not be repeated here.

[0090] In some embodiments, a computer-readable storage medium is provided on which a computer program is stored. In different embodiments, when the computer program is executed by a processor, the steps performed by each functional module or functional unit of the device for acquiring depressive symptom characteristics based on facial expression changes described in the previous embodiments are respectively implemented.

[0091] Those skilled in the art will appreciate that all or part of the process of implementing the method for acquiring depressive symptom characteristics based on facial expression changes can be accomplished by instructing related hardware through a computer program, and the aforementioned computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it may include the process of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0092] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to connect to an external terminal to receive a video stream of the user's facial expression uploaded by the terminal. When the computer program is executed by the processor, a method for acquiring depressive symptom features based on changes in facial expressions is implemented.

[0093] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0094] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented: obtaining a video stream of facial expressions of a target user, wherein the video stream includes multiple changing facial expressions of the target user; obtaining the number of video frames in the video stream and facial expression features of each frame in the video stream, and obtaining a first feature tensor based on the number of video frames and the facial expression features of each frame; inputting the first feature tensor into a pre-trained self-attention model to obtain a second feature vector output by the pre-trained self-attention model, wherein the second feature vector represents the depressive symptom features of the target user, and the pre-trained self-attention model is used to determine the importance of the facial expression features of each frame of the video corresponding to the depressive state based on the first feature tensor and output a second feature vector based on the importance.

[0095] In one embodiment, when the processor executes the computer program, the following steps are further implemented: determining a third eigenvector according to S0=Q*K0, wherein S0 represents the third eigenvector, the third eigenvector is a vector representing the importance of each frame of video in the first eigentensor, Q represents the first eigentensor, and K0 is a representation vector representing the difference in facial expressions of corresponding video frames in the video stream in distinguishing between users with depression and healthy users; determining position information of n values ​​of S0 ranked in the top n positions according to Pos=argTopn(S0), Pos represents the position information of n values ​​of S0 ranked in the top n positions, and Topn represents the position information of n values ​​of S0 ranked in the top n positions. values, arg represents the position information of the extracted n values, and n represents a positive integer; according to Q1=Q[Pos,:], the fourth eigenvector is determined, Q1 represents the fourth eigenvector, Q[Pos,:] represents the eigenvector obtained by extracting the elements of Pos from Q and corresponding to the elements of each row in Q and combining the elements of each row; according to S1=softmax(Q1*K1), the importance of each element in the fourth eigenvector is determined, S1 represents the importance of each element in the fourth eigenvector, K1 represents the representation vector of the facial expression difference of the corresponding video frame in Q1 in distinguishing between depression users and healthy users, and softmax represents the softmax function; according to P=S1 T *Q1 determines the depressive symptom characteristics, and P represents the second eigenvector.

[0096] In one embodiment, when the processor executes the computer program, the following steps are further implemented: when the first feature tensor is input into a pre-trained self-attention model, a feature vector of the importance of each frame of video in the video stream output by the pre-trained self-attention model relative to the second feature vector is obtained; wherein the self-attention model obtains the feature vector of the importance of each frame of video relative to the second feature vector according to C=softmax(S0), and C represents the feature vector of the importance of each frame of video relative to the second feature vector.

[0097] In one embodiment, when the processor executes the computer program, it further implements the following steps: obtaining video data of facial expressions of a user with depression and video data of facial expressions of a healthy user; randomly initializing K0 and K1; inputting the video data of facial expressions of the user with depression and video data of facial expressions of the healthy user into a self-attention model, and obtaining a pre-trained self-attention model by calculating the cross-entropy loss and updating K0 and K1 using a deep learning gradient update algorithm and a back-propagation algorithm.

[0098] In one embodiment, when the processor executes a computer program to implement the above-mentioned steps of obtaining the number of video frames in the video stream and the facial expression features of each frame in the video stream and obtaining a first feature tensor based on the number of video frames and the facial expression features of each frame, the following steps are specifically implemented: inputting the video stream into a pre-trained convolutional neural network to obtain a feature vector of the facial expression features of each frame in the video stream output by the pre-trained convolutional neural network; and splicing the feature vectors of the facial expression features of each frame in the video stream side by side according to the number of video frames to obtain a first feature tensor.

[0099] In one embodiment, when the processor executes the computer program, it further implements the following steps: obtaining a picture set, inputting the picture set into a convolutional neural network to obtain a preliminary convolutional neural network, where the preliminary convolutional neural network is a network model that can represent the abstract content of the picture; obtaining pictures of various types of facial expressions, inputting the pictures of various types of facial expressions into the preliminary convolutional neural network to obtain a pre-trained convolutional neural network.

[0100] In one embodiment, when the processor executes the computer program, it further implements the following steps: inputting the second eigenvector into a pre-trained multi-layer perceptron to obtain an evaluation result of the depressive expression output by the pre-trained multi-layer perceptron; wherein the pre-trained multi-layer perceptron is trained using facial expression data of users with depression and facial expression data of healthy users.

[0101] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining a video stream of facial expressions of a target user, the video stream containing multiple changing facial expressions of the target user; obtaining the number of video frames in the video stream and the facial expression features of each frame of video in the video stream and obtaining a first feature tensor based on the number of video frames and the facial expression features of each frame of video; inputting the first feature tensor into a pre-trained self-attention model to obtain a second feature vector output by the pre-trained self-attention model, the second feature vector representing the depressive symptom features of the target user, and the pre-trained self-attention model is used to determine the importance of the facial expression features of each frame of video corresponding to the depressive state based on the first feature tensor and output a second feature vector based on the importance.

[0102] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining a third eigenvector according to S0=Q*K0, wherein S0 represents the third eigenvector, the third eigenvector is a vector representing the importance of each frame of video in the first eigentensor, Q represents the first eigentensor, and K0 is a representation vector representing the difference in facial expressions of corresponding video frames in the video stream in distinguishing between users with depression and healthy users; determining position information of n values ​​of S0 ranked in the top n positions according to Pos=argTopn(S0), Pos represents the position information of n values ​​of S0 ranked in the top n positions, and Topn represents the position information of n values ​​of S0 ranked in the top n positions. values, arg represents the position information of the extracted n values, and n represents a positive integer; according to Q1=Q[Pos,:], the fourth eigenvector is determined, Q1 represents the fourth eigenvector, Q[Pos,:] represents the eigenvector obtained by extracting the elements of Pos from Q and corresponding to the elements of each row in Q and combining the elements of each row; according to S1=softmax(Q1*K1), the importance of each element in the fourth eigenvector is determined, S1 represents the importance of each element in the fourth eigenvector, K1 represents the representation vector of the facial expression difference of the corresponding video frame in Q1 in distinguishing between depression users and healthy users, and softmax represents the softmax function; according to P=S1 T *Q1 determines the depressive symptom characteristics, and P represents the second eigenvector.

[0103] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when the first feature tensor is input into a pre-trained self-attention model, a feature vector of the importance of each frame of video in the video stream output by the pre-trained self-attention model relative to the second feature vector is obtained; wherein the self-attention model obtains the feature vector of the importance of each frame of video relative to the second feature vector according to C=softmax(S0), and C represents the feature vector of the importance of each frame of video relative to the second feature vector.

[0104] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: obtaining video data of facial expressions of a user with depression and video data of facial expressions of a healthy user; randomly initializing K0 and K1; inputting the video data of facial expressions of the user with depression and video data of facial expressions of the healthy user into a self-attention model, and updating K0 and K1 by calculating the cross-entropy loss and using the gradient update algorithm of deep learning and the back-propagation algorithm to obtain a pre-trained self-attention model.

[0105] In one embodiment, when the computer program is executed by the processor to implement the above-mentioned steps of obtaining the number of video frames in the video stream and the facial expression features of each frame of the video stream and obtaining the first feature tensor based on the number of video frames and the facial expression features of each frame of the video, the following steps are specifically implemented: inputting the video stream into a pre-trained convolutional neural network to obtain the feature vectors of the facial expression features of each frame of the video stream output by the pre-trained convolutional neural network; and splicing the feature vectors of the facial expression features of each frame of the video stream side by side according to the number of video frames to obtain the first feature tensor.

[0106] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: obtaining a set of pictures, inputting the set of pictures into a convolutional neural network to obtain a preliminary convolutional neural network, where the preliminary convolutional neural network is a network model that can represent the abstract content of the pictures; obtaining pictures of various types of facial expressions, inputting the pictures of various types of facial expressions into the preliminary convolutional neural network to obtain a pre-trained convolutional neural network.

[0107] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: inputting the second eigenvector into a pre-trained multi-layer perceptron to obtain an evaluation result of the depressive expression output by the pre-trained multi-layer perceptron; wherein the pre-trained multi-layer perceptron is trained using facial expression data of users with depression and facial expression data of healthy users.

[0108] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0109] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0110] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A device for acquiring depressive symptom features based on facial expression changes, the device comprising: A video stream acquisition module is used to acquire a video stream of facial expressions of a target user, wherein the video stream includes a plurality of changing facial expressions of the target user; A first feature tensor acquisition module is used to obtain the number of video frames in the video stream and the facial expression features of each frame of video in the video stream, and obtain a first feature tensor according to the number of video frames and the facial expression features of each frame of video; a second feature vector acquisition module, configured to input the first feature tensor into a pre-trained self-attention model to obtain a second feature vector output by the pre-trained self-attention model, wherein the second feature vector represents the depressive symptom characteristics of the target user, and the pre-trained self-attention model is configured to determine, based on the first feature tensor, the importance of the facial expression features of each frame of the video corresponding to the depressive state and output the second feature vector based on the importance; A self-attention model construction module, wherein the self-attention model construction module comprises: The third feature vector acquisition unit is used to obtain the feature vector according to Determine the third eigenvector, where represents the third eigenvector, which is a vector representing the importance of each frame of video in the first eigentensor. represents the first eigentensor, A representation vector representing the difference in facial expressions between a depressed user and a healthy user in a corresponding video frame in the video stream; Position information acquisition unit, used to Sure Median size comes first Position The location information of the value, express Median size comes first Position The location information of the value, Representation extraction Median size comes first Position values, Indicates the extracted The location information of the value, represents a positive integer; The fourth eigenvector obtaining unit is used to obtain the Determine the fourth eigenvector, represents the fourth eigenvector, Indicates from Extract The elements of and combining the elements of each row in the eigenvector obtained by combining the elements of each row; Importance acquisition unit, used to Determine the importance of each element in the fourth eigenvector, represents the importance of each element in the fourth eigenvector, To express The representation vector of the facial expression difference between the corresponding video frame in distinguishing depression users and healthy users, express function; Depression symptom feature acquisition unit, used to Determining the depressive symptom characteristics, represents the second eigenvector; A training module of a self-attention model, wherein the training module of the self-attention model comprises: a training data acquisition unit, configured to acquire video data of facial expressions of users with depression and video data of facial expressions of healthy users; Random value setting unit, used to randomly initialize the and stated ; The model training unit is configured to input the video data of the facial expression of the depressed user and the video data of the facial expression of the healthy user into the self-attention model, and update the self-attention model by calculating the cross entropy loss and using the gradient update algorithm of deep learning and the back propagation algorithm. and stated , and obtain the pre-trained self-attention model.

2. The device according to claim 1, characterized in that The device further comprises: a contribution vector acquisition module, configured to obtain, when the first feature tensor is input into a pre-trained self-attention model, a feature vector of the importance of each frame of video in the video stream output by the pre-trained self-attention model relative to the second feature vector; The self-attention model is based on Obtaining a feature vector of the importance of each frame of video relative to the second feature vector, A feature vector representing the importance of each frame of video relative to the second feature vector.

3. The device according to claim 1, characterized in that The first feature tensor acquisition module includes: A facial expression feature acquisition unit is used to input the video stream into a pre-trained convolutional neural network to obtain a feature vector of the facial expression feature of each frame of video in the video stream output by the pre-trained convolutional neural network; The first feature tensor acquisition unit is used to splice the feature vectors of the facial expression features of each frame of video in the video stream side by side according to the number of video frames to obtain the first feature tensor.

4. The device according to claim 3, characterized in that The apparatus further includes a convolutional neural network training module, wherein the convolutional neural network training module includes: A first training unit is configured to obtain a set of images and input the set of images into a convolutional neural network to obtain a preliminary convolutional neural network, wherein the preliminary convolutional neural network is a network model capable of representing abstract content of the images; The second training unit is used to obtain various types of facial expression pictures, and input the various types of facial expression pictures into the preliminary convolutional neural network to obtain the pre-trained convolutional neural network.

5. The device according to claim 1, characterized in that The device further comprises: a depressive expression analysis module, configured to input the second feature vector into a pre-trained multi-layer perceptron to obtain a depressive expression evaluation result output by the pre-trained multi-layer perceptron; The pre-trained multi-layer perceptron is trained using facial expression data of depression users and facial expression data of healthy users.

6. A system for acquiring depressive symptom features based on facial expression changes, characterized in that: Including servers and terminals; The terminal is used to collect a video stream of facial expressions of a target user and send the video stream to the server, wherein the video stream includes multiple facial expressions of the target user; The server is used to obtain the video stream sent by the terminal, obtain the number of video frames in the video stream and the facial expression features of each frame in the video stream, and obtain a first feature tensor based on the number of video frames and the facial expression features of each frame, input the first feature tensor into a pre-trained self-attention model to obtain a second feature vector output by the pre-trained self-attention model, the second feature vector characterizing the depressive symptom features of the target user, and the pre-trained self-attention model is used to determine the importance of the facial expression features of each frame of video corresponding to the depressive state based on the first feature tensor and output the second feature vector based on the importance; The server is also used to Determine the third eigenvector, where represents the third eigenvector, which is a vector representing the importance of each frame of video in the first eigentensor. represents the first eigentensor, is a representation vector representing the difference in facial expressions between the corresponding video frame in the video stream and the healthy user; according to Sure Median size comes first Position The location information of the value, express Median size comes first Position The location information of the value, Representation extraction Median size comes first Position values, Indicates the extracted The location information of the value, Represents a positive integer; according to Determine the fourth eigenvector, represents the fourth eigenvector, Indicates from Extract The elements of The characteristic vector obtained by combining the elements of each row in Determine the importance of each element in the fourth eigenvector, represents the importance of each element in the fourth eigenvector, To express The representation vector of the facial expression difference between the corresponding video frame in distinguishing depression users and healthy users, express function; according to Determining the depressive symptom characteristics, represents the second eigenvector; The server is also used to obtain video data of facial expressions of depression users and video data of facial expressions of healthy users; randomly initialize the and stated The video data of the facial expression of the depression user and the video data of the facial expression of the healthy user are input into the self-attention model, and the cross entropy loss is calculated and the gradient update algorithm of deep learning and the back propagation algorithm are used to update the self-attention model. and stated , and obtain the pre-trained self-attention model.

7. A method for acquiring depressive symptom features based on facial expression changes, characterized in that: The method comprises: Obtaining a video stream of facial expressions of a target user, wherein the video stream includes multiple changing facial expressions of the target user; Obtaining the number of video frames of the video stream and the facial expression features of each frame of video in the video stream and obtaining a first feature tensor according to the number of video frames and the facial expression features of each frame of video; Inputting the first feature tensor into a pre-trained self-attention model to obtain a second feature vector output by the pre-trained self-attention model, wherein the second feature vector represents the depressive symptom characteristics of the target user, and the pre-trained self-attention model is used to determine the importance of the facial expression features of each frame of the video corresponding to the depressive state based on the first feature tensor and output the second feature vector based on the importance; The step of inputting the first feature tensor into a pre-trained self-attention model to obtain a second feature vector output by the pre-trained self-attention model comprises: Determine the third eigenvector, where represents the third eigenvector, which is a vector representing the importance of each frame of video in the first eigentensor. represents the first eigentensor, is a representation vector representing the difference in facial expressions between the corresponding video frame in the video stream and the healthy user; according to Sure Median size comes first Position The location information of the value, express Median size comes first Position The location information of the value, Representation extraction Median size comes first Position values, Indicates the extracted The location information of the value, Represents a positive integer; according to Determine the fourth eigenvector, represents the fourth eigenvector, Indicates from Extract The elements of The characteristic vector obtained by combining the elements of each row in Determine the importance of each element in the fourth eigenvector, represents the importance of each element in the fourth eigenvector, To express The representation vector of the facial expression difference between the corresponding video frame in distinguishing depression users and healthy users, express function; according to Determining the depressive symptom characteristics, represents the second eigenvector; The training steps of the self-attention model include: obtaining video data of facial expressions of depression users and video data of facial expressions of healthy users; randomly initializing the and stated The video data of the facial expression of the depression user and the video data of the facial expression of the healthy user are input into the self-attention model, and the cross entropy loss is calculated and the gradient update algorithm of deep learning and the back propagation algorithm are used to update the self-attention model. and stated , and obtain the pre-trained self-attention model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 7 are implemented.

Images