Mental disease identification method and device based on computer vision

Abstract

The invention provides a mental disease identification method and device based on computer vision. The method comprises the following steps: step 1), collecting video data; step 2), processing the video data: extracting facial feature points on the basis of the step 1), and performing smooth filtering and normalization operation on facial mark points to reduce noise of the data and simplify calculation; 3) constructing distance features and position features; 4) constructing a facial space change model of the mental patient; step 5), designing a triple facial feature flow shallow neural network for training; and 6) outputting an identification result. According to the method disclosed by the invention, mental disease identification based on computer vision is realized under the condition that a user inquiry video with good definition exists. Experimental results show that the method can effectively identify the mental diseases by analyzing the visual features in the video, and a new technical means is provided for auxiliary diagnosis of the mental diseases.

Description

Technical Field

[0001] The present invention relates to the application of computer vision technology in the field of medical diagnosis, and in particular to a method and device for identifying mental illness based on computer vision. The method uses computer vision technology to analyze changes in facial expressions of patients to achieve automatic identification and discrimination of mental illness. Background Art

[0002] As the pace of modern life accelerates, people feel pressure from all aspects. The accumulation of pressure leads to various mental health problems. These mental illnesses harm people's body and mind, and in severe cases even cause patients to behave in a way that threatens the life safety of themselves or others.

[0004] Detection of mental illness has always been a complex and time-consuming task. To determine whether a person is suffering from mental illness, one must not only have professional knowledge and basic personal information, but also observe the target population over a period of time for clinical diagnosis. Many people with mild mental illnesses are often unable to realize the abnormality of their mental state, and will not take the initiative to disclose their own conditions or seek treatment, which leads to huge practical difficulties in detecting mental health problems. Minor psychological abnormalities are likely to accumulate over time and lead to irreparable consequences, and psychological or psychiatric diagnosis after a tragedy is obviously too late.

[0005] In recent years, computer vision technology has been increasingly used in the field of medical diagnosis, especially in the diagnosis of mental illness. This technology can objectively and continuously record and analyze the patient's behavior by analyzing the patient's facial expressions, eye movements and other visual features, providing new possibilities for early diagnosis and continuous monitoring of mental illness.

[0006] At the same time, the development of emotion recognition technology has promoted a new hot spot in the research of mental illness identification. Researchers collect multimodal indicators, such as behavioral signals (eye movements, gait, voice and facial expressions, etc.) and physiological signals (electroencephalogram, nuclear magnetic resonance, mental images, etc.), and combine advanced technologies such as statistical analysis and deep learning to build models to identify the differences between patients with mental illness and healthy people. Among these indicators, facial expression analysis based on computer vision technology has become a very promising direction in the diagnosis of mental illness due to its non-invasiveness and easy access.

[0007] Compared with other behavioral and physiological signals, facial expressions have the advantages of being non-intrusive, easy to obtain, and low-cost in the collection process. Facial expressions can be used to study the psychological activities, emotional changes, and mental states of patients with mental illness. The clinical manifestations of patients with mental illness are usually characterized by reduced facial expression richness, drooping eyes, frowning, drooping corners of the mouth, less smiles, and easy crying. According to the Mehrabin rule, emotional communication is manifested in 55% facial expressions, 38% tone and body movements, and 7% language. It can be seen that in the study of mental illness identification, modeling through facial expression information analysis has become a very effective means.

[0008] The application of computer vision technology in facial expression analysis further enhances the advantages of this method. Through high-precision image capture and advanced image processing algorithms, computer vision systems can accurately locate and track facial feature points and analyze subtle changes in expression, thus providing richer and more accurate data support for the diagnosis of mental illness.

[0009] Facial expression is essentially a dynamic change process. Therefore, dynamic image sequences contain more facial context information than single static images and can better describe facial features. The present invention studies dynamic image sequences of facial videos and constructs a method for identifying expression changes for mental illness. Summary of the invention

[0010] The present invention aims to solve the above-mentioned deficiencies of existing computer vision-based mental illness detection technology and provide a method and device for identifying facial expression changes in mental illness.

[0011] Based on expression analysis technology, the present invention starts from the dynamic change characteristics of facial landmarks, studies the weights and influence of distance features and position features constructed by facial landmarks in different parts, uses the proposed ratio vector mapping function to solve the individual differences of different patients, and uses patient consultation videos to realize a high-accuracy and high-efficiency expression change recognition method for mental illness.

[0012] In order to achieve the above object, the present invention provides a method for identifying facial expression changes for mental illness, comprising the following steps:

[0013] Step 1) Preprocessing of video data of consultation with mental illness experts, including:

[0014] (1.1) Organize the consultation videos of mental illness experts and extract the core clips of the videos based on the consultation records;

[0015] (1.2) Use the facial feature point extraction tool on the market to extract the feature points of the faces in the medical consultation video clips.

[0016] Obtain 68 facial three-dimensional coordinate point data sorted in time sequence;

[0017] (1.3) Based on the 68 facial landmarks, distance and position feature vectors are constructed. The facial landmarks are first smoothed and normalized to reduce data noise and simplify calculations.

[0018] (1.4) According to the clinical research results, based on the above feature points, several distance features and position features are constructed. The distance features constructed based on facial landmarks are: the distance between the left eyebrow and the nose D1, the distance between the right eyebrow and the nose D2, the distance between the left and right eyebrows D3, the distance between the left eye and the left corner of the mouth D4, the distance between the right eye and the right corner of the mouth D5, the distance between the left eye and the left side of the nose D6, the distance between the right eye and the right side of the nose D7, the distance between the left side of the nose and the left corner of the mouth D8, and the distance between the right side of the nose and the right corner of the mouth D9; the definition and meaning of the distance features of facial landmarks are shown in Table (1-1);

[0019] Table 1-1 Definition and meaning of facial landmark distance features

[0020]

[0021] (1.5) The specific definitions of the position features constructed based on facial landmarks are shown in formulas (2-1), (2-2), (2-3), (2-4), (2-5), (2-6), and (2-7). Their specific meanings are the degree of opening and closing of the left eye L1, the degree of opening and closing of the right eye L2, the degree of opening and closing of the inner lip L3, the degree of opening and closing of the outer lip L4, the degree of curvature of the left eyebrow L5, the degree of curvature of the right eyebrow L6, and the distance from the nose tip to the upper lip L7;

[0022]

[0023] (1.6) Perform secondary feature construction on the constructed distance features and position features according to the time series to form a standard feature vector of a video. At the same time, calculate its statistical features including maximum value, minimum value, mean, median, mode, range, standard deviation, skewness, kurtosis and quartile range;

[0024] (1.7) Calculate the velocity and acceleration features from the distance features and position features, and obtain the dynamic time series corresponding to the two types of features. Similarly, extract the statistical features from them to complete the preliminary feature extraction;

[0025] Step 2) Derivation of theoretical basis for feature extraction and elimination of individual differences, specifically including:

[0026] (2.1) The distance characteristics of facial features vary greatly among different individuals. Therefore, a constant distance is selected for each person to eliminate individual differences, so that these features do not depend on the different facial structures and facial areas of different people, nor on the individual differences in facial topology of the same facial expression;

[0027] (2.2) Analyze the spatial transformation that occurs during the patient consultation, including the change in the depth of the face from the camera lens, the movement of the face, the rotation and deflection of the face, so as to construct a facial spatial change model for patients with mental illness;

[0028] (2.3) Based on the facial spatial change model of patients with mental illness, a ratio vector mapping function is constructed. The ratio vector mapping function is shown in (2-8). Among them, k1, k2 and k3 are the slopes of J1J2, J3J4 and Y1Y2 respectively, and S(Y1Y2) is the ratio vector mapping function;

[0029]

[0030] (2.4) adding the features extracted in step (1.7) to the ratio vector mapping function to enhance the anti-interference and anti-attack capabilities of the extracted features;

[0031] Step 3) Build a mental illness classification model, including:

[0032] (3.1) A triple facial feature stream shallow neural network is designed for the classification of mental illness. For a long mental illness consultation video, it is first divided into short video clips of equal length according to the dialogue or the offset of the landmarks, and the redundant video clips are discarded. Three groups of feature vectors are extracted from the divided short video clips in time sequence, and spliced ​​according to the facial structure to obtain the position feature vector map, distance feature vector map and dynamic feature vector map;

[0033] (3.2) Input the three sets of feature vectors into the triple facial feature stream shallow neural network. At the beginning of the model, extract the features, perform a convolution and a maximum pooling to maintain the independence of each set of features, then concatenate the high-dimensional features, perform a maximum pooling again, and finally connect to the fully connected layer to output a score value;

[0034] (3.3) In the design of the triple facial feature stream shallow neural network, the classification task of a single long video is transformed into a regression task of multiple short videos. Each short video clip needs to be pseudo-labeled through the Heaviside step function for training to obtain a weight model;

[0035] Step 4) Identify the patient’s mental illness, including:

[0036] (4.1) Select the patient consultation video to be diagnosed, extract the feature point data, cut it according to the facial landmark point offset, and convert the data into short time series unit data;

[0037] (4.2) Load the triple facial feature stream shallow neural network model saved in step 3, input the short time sequence unit data obtained in (4.1), and obtain the model output scores of multiple short time sequence units;

[0038] (4.3) Aggregate the scores of the multiple short time sequence units obtained in step 3 to obtain the overall score;

[0039] (4.4) Based on the overall score obtained in (4.3), output the patient’s mental illness category;

[0040] The second aspect of the present invention relates to a computer vision-based mental illness identification device, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a computer vision-based mental illness identification method of the present invention.

[0041] A computer-readable storage medium of the present invention stores a program, which, when executed by a processor, implements a computer vision-based mental illness identification method of the present invention.

[0042] The method of the present invention realizes the recognition of mental illness based on computer vision under the condition of having a user consultation video with good definition. The experimental results show that the present invention can effectively identify mental illness by analyzing the visual features in the video, and provides a new technical means for the auxiliary diagnosis of mental illness.

[0043] The advantages of the present invention are: the method of the present invention is easy to operate, the model has a shallow number of layers, the training is fast, and the matching accuracy is high. Since in the current diagnosis of mental illness, it is necessary to go to a professional medical institution for diagnosis, the diagnosis cycle is long, the procedures are cumbersome, and the cost is high. Therefore, the present invention can be used as an auxiliary means for mental illness detection. The user only needs to record the consultation video through the camera and upload it to the model to get the result. The feedback speed is fast, which provides a reference for the user whether to choose to go to the hospital for treatment later. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of well-collected facial feature points and distance features of the present invention;

[0045] Figure 2 A schematic diagram of a patient's facial landmark spatial variation model of the present invention;

[0046] Figure 3 Schematic diagram of the triple facial feature stream shallow neural network of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings.

[0048] Example 1

[0049] This embodiment relates to a method for identifying mental illness based on computer vision, comprising the following steps:

[0050] Step 1) Preprocessing of video data of consultation with mental illness experts, including:

[0051] (1.1) Organize the consultation videos of mental illness experts and extract the core clips of the videos based on the consultation records;

[0052] (1.2) Use a facial feature point extraction tool available on the market to extract feature points from the face in the medical consultation video clip to obtain 68 facial landmark point coordinates, as shown in Table 1-1;

[0053] Table 1-1 Example of facial landmark coordinates

[0054]

[0055]

[0056]

[0057] (1.3) Based on the 68 facial landmarks, distance and position feature vectors are constructed. The facial landmarks are first smoothed and normalized. The smoothing filter uses the Gaussian filter method, and the normalization operation uses the L2 normalization method to reduce data noise and simplify calculations.

[0058] (1.4) According to the clinical research results, based on the above-mentioned feature points, several distance features and position features are constructed. The meaning of the distance features constructed based on the facial landmark points is: the distance between the left eyebrow and the nose D1, the distance between the right eyebrow and the nose D2, the distance between the left and right eyebrows D3, the distance between the left eye and the left corner of the mouth D4, the distance between the right eye and the right corner of the mouth D5, the distance between the left eye and the left side of the nose D6, the distance between the right eye and the right side of the nose D7, the distance between the left side of the nose and the left corner of the mouth D8, and the distance between the right side of the nose and the right corner of the mouth D9; when this embodiment is implemented, the definition and meaning of the facial landmark point distance features are shown in Table 1-2 below:

[0059] Table 1-2 Definition and meaning of facial landmark distance features

[0060]

[0061] (1.5) The specific definitions of the position features constructed based on facial landmarks are as follows:

[0062] (2-5), (2-6), and (2-7). Their specific meanings are respectively the degree of opening and closing of the left eye L1, the degree of opening and closing of the right eye L2, the degree of opening and closing of the inner lip L3, the degree of opening and closing of the outer lip L4, the degree of curvature of the left eyebrow L5, the degree of curvature of the right eyebrow L6, and the distance from the nose tip to the upper lip L7, where dis means calculating the distance between the coordinates of two points;

[0063]

[0064]

[0065] (1.6) The constructed distance features and position features are recalculated according to the time series. The specific method is to perform polynomial expansion on the original features to obtain a higher-dimensional feature vector, which is the standard feature vector of a video. At the same time, its statistical features are calculated, including the maximum value, minimum value, mean, median, mode, range, standard deviation, skewness, kurtosis and quartile range, maximum value, minimum value, mean, median, mode. These statistics can be obtained by directly calculating each column of the feature vector. The range is the difference between the maximum and minimum values, which can be obtained by calculating the difference between the maximum and minimum values ​​of each column of the feature vector. The standard deviation can be obtained by calculating the mean and variance of each column. The skewness and kurtosis can be obtained by calculating the skewness and kurtosis coefficients of each column.

[0066] (1.7) By using basic physical formulas to obtain velocity and acceleration features for distance features and position features, we can obtain the dynamic time series corresponding to the two types of features. If we need to obtain velocity and acceleration features for distance features and position features, we can use basic mathematical formulas to calculate them. Similarly, we can extract statistical features from them and complete the preliminary feature extraction according to the method in Section 1.6.

[0067] Step 2) Derivation of theoretical basis for feature extraction and elimination of individual differences, specifically including:

[0068] (2.1) The distance features of the facial features of a person vary greatly among different individuals. Therefore, a constant distance is selected for each person to eliminate individual differences. For example, the distance from the center of the eye to the tip of the nose, the distance from the corner of the mouth to the earlobe, the distance from the inner side of the eyebrow to the corner of the eye, and the distance from the nose to the ear can be selected. For each person's distance feature, it can be compared with the distance features of all people, and the ranking percentage of the distance feature among all people can be calculated. Then, the percentage can be used as the distance feature value of the person to eliminate individual differences. These features do not depend on the different facial structures and facial areas of different people, nor do they depend too much on the individual differences in facial topology of the same facial expression;

[0069] (2.2) Analyze the spatial transformation that occurs during the patient consultation, which includes the change in the depth of the face from the camera lens, the position movement of the face, the rotation and deflection of the face, so as to construct a facial spatial change model for patients with mental illness. The steps are as follows: collect a large number of facial images and related medical records of patients with different mental illnesses, including disease type, course of disease, treatment plan, etc. At the same time, it is also necessary to collect some facial images of healthy people as a control group, annotate the collected facial images, including the change in the depth of the face from the camera lens, the position movement of the face, the rotation and deflection of the face, and use computer vision technology to extract features from the annotated facial images, and extract features related to spatial transformation, such as the position coordinates of the face, rotation angle, deflection angle, etc.

[0070] (2.3) Based on the facial spatial change model of patients with mental illness, a ratio vector mapping function is constructed. The ratio vector mapping function is shown in (2-8). Among them, k1, k2 and k3 are the slopes of J1J2, J3J4 and Y1Y2 respectively, and S(Y1Y2) is the ratio vector mapping function;

[0071]

[0072] (2.4) adding the features extracted in step (1.7) to the ratio vector mapping function to enhance the anti-interference and anti-attack capabilities of the extracted features;

[0073] Step 3) Build a mental illness classification model, including:

[0074] (3.1) A triple facial feature stream shallow neural network is designed for the classification of mental illness. For a long mental illness consultation video as input, it is first divided into short video clips of equal length according to the dialogue or the offset of the landmarks, and the redundant video clips are discarded. Three groups of feature vectors are extracted from the divided short video clips in time sequence and spliced ​​according to the facial structure. The specific steps of splicing according to the facial structure to obtain the three groups of feature vectors are as follows: according to the predefined facial key points (such as eyes, nose, mouth, etc.), the position coordinates of each key point in the image are extracted, and the position coordinates of all the key points are connected in a predefined order to form a position feature vector, and the distance feature vector graph is calculated according to the predefined facial key points. The distance between any two key points is connected in a predefined order to form a distance feature vector, and the dynamic feature vector graph is According to the predefined facial key points, the change rate of each key point in time is calculated, and all the change rates are connected in a predefined order to form a dynamic feature vector, wherein the predefined order is as follows: inner side of left eyebrow, left eyebrow peak, outer side of left eyebrow, inner side of right eyebrow, right eyebrow peak, outer side of right eyebrow, upper left corner of left eye, upper right corner of left eye, lower right corner of left eye, lower left corner of left eye, upper left corner of right eye, upper right corner of right eye, lower right corner of right eye, lower left corner of right eye, upper left part of nose, top of nose, upper right part of nose, lower left part of nose, lower right part of nose, upper left corner of mouth, upper right corner of mouth, lower right corner of mouth, lower left corner of mouth;

[0075] (3.2) The three sets of feature vector images are input into the triple facial feature stream shallow neural network. At the beginning of the neural network model, the features are extracted by splitting, and three sets of separate convolution kernels are used to calculate the input feature vector images. One convolution and one maximum pooling are performed to maintain the independence of each set of features. Then, the results of the three sets of convolution outputs are superimposed on the number of channels to obtain high-dimensional features, and then the maximum pooling is performed again. Finally, they are sent to the fully connected layer of the neural network, and an activation function is used to output a score, where convolution, maximum pooling, fully connected layers and activation functions are all general computer technologies;

[0076] (3.3) In the design of the triple facial feature stream shallow neural network, the classification task of a single long video is transformed into a regression task of multiple short videos. Each short video clip needs to be pseudo-labeled through the Heaviside step function for training to obtain the weight model derived from the neural network. The pseudo-labeling method here refers to labeling each short video clip according to whether it belongs to the target category, that is, marking it as 1 or 0;

[0077] Step 4) Identify the patient’s mental illness, including:

[0078] (4.1) Select the patient consultation video to be diagnosed, extract the feature point data, cut it according to the facial landmark offset, and convert the data into short time sequence unit data. The specific steps are as follows: select the patient consultation video to be diagnosed, and use the facial feature point detection algorithm (such as dlib or OpenCV, etc.) to extract the facial feature point coordinates of each frame in the video. According to the facial landmark offset, cut the facial feature point sequence of each frame into several subsequences of equal length, for example, cut the entire facial feature point sequence into 10 subsequences of length 100. For each subsequence, regard it as a short time sequence unit data and save it in a new data set. Repeat the above steps until all facial feature point sequences are cut and converted into short time sequence unit data;

[0079] (4.2) Use PyTorch to load the triple facial feature stream shallow neural network model saved in step 3, input the short time sequence unit data obtained in (4.1), and after the neural network code calculation, obtain the model output scores of multiple short time sequence units;

[0080] (4.3) Aggregate the scores of the multiple short time series units obtained in step 3 to obtain an overall score. Specifically, for each short time series unit, calculate the probability distribution of its belonging to each mental illness category. For example, a general softmax function can be used to convert the original score output by the model into a probability distribution. For all short time series units, calculate their average probability values ​​belonging to each mental illness category, that is, average the probability values ​​of a certain category of all short time series units to obtain the overall score of the category;

[0081] (4.4) Based on the overall score obtained in (4.3), output the patient's mental illness category according to the statistical relationship between the score and the disease type;

[0082] This embodiment provides a computer vision-based method for identifying mental illness. It mainly uses the expression change recognition method. According to the weights and influence of distance features and position features constructed by facial feature points in different parts, 9 groups of distance features and 7 groups of position features are finally selected. The proposed ratio vector mapping function is used to solve the individual differences of different patients, and the patient consultation video is used to achieve high-accuracy and high-efficiency mental illness detection.

[0083] Example 2

[0084] This embodiment relates to a computer vision-based mental illness identification device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a computer vision-based mental illness identification method of Example 1.

[0085] Example 3

[0086] This embodiment is a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a computer vision-based mental illness identification method according to embodiment 1 is implemented.

[0087] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for identifying mental illness based on computer vision, comprising the following steps: Step 1), pre-processing the video data of the consultation with the mental illness expert; Step 2), derive the theoretical basis of feature extraction and eliminate individual differences; Step 3), build a mental illness classification model; Step 4) Identify the patient's mental illness category.

2. A computer vision-based mental illness identification method as claimed in claim 1, characterized in that: Step 1) Specifically include: (1.1) Organize the consultation videos of mental illness experts and extract the core clips of the videos based on the consultation records; (1.2) Using a facial feature point extraction tool available on the market, extract the feature points of the face in the medical consultation video clip to obtain 68 facial three-dimensional coordinate point data sorted in time sequence; (1.3) Based on the 68 facial landmarks, distance and position feature vectors are constructed. The facial landmarks are first smoothed and normalized to reduce data noise and simplify calculations. (1.4) According to the clinical research results, based on the above feature points, several distance features and position features are constructed. The meaning of the distance features constructed based on facial landmarks is shown in the following table (1-1); Table 1-1 Definition and meaning of facial landmark distance features (1.5) The specific definitions of the position features constructed based on facial landmarks are shown in formulas (2-1), (2-2), (2-3), (2-4), (2-5), (2-6), and (2-7). Their specific meanings are the degree of opening and closing of the left eye L1, the degree of opening and closing of the right eye L2, the degree of opening and closing of the inner lip L3, the degree of opening and closing of the outer lip L4, the degree of curvature of the left eyebrow L5, the degree of curvature of the right eyebrow L6, and the distance from the nose tip to the upper lip L7; (1.6) Perform secondary feature construction on the constructed distance features and position features according to the time series to form a standard feature vector of a video. At the same time, calculate its statistical features including maximum value, minimum value, mean, median, mode, range, standard deviation, skewness, kurtosis and quartile range; (1.7) The velocity and acceleration features are obtained from the distance features and position features to obtain the dynamic time series corresponding to the two types of features. The statistical features are also extracted to complete the preliminary feature extraction.

3. A method for identifying mental illness based on computer vision as claimed in claim 1, characterized in that: The theoretical basis derivation and individual difference elimination of feature extraction described in step 2) specifically include: (2.1) The distance characteristics of facial features vary greatly among different individuals. Therefore, a constant distance is selected for each person to eliminate individual differences, so that these features do not depend on the different facial structures and facial areas of different people, nor on the individual differences in facial topology of the same facial expression; (2.2) Analyze the spatial transformation that occurs during the patient consultation, including the change in the depth of the face from the camera lens, the movement of the face, the rotation and deflection of the face, so as to construct a facial spatial change model for patients with mental illness; (2.3) Based on the facial spatial change model of patients with mental illness, a ratio vector mapping function is constructed. The ratio vector mapping function is shown in (2-8). Among them, k1, k2 and k3 are the slopes of J1J2, J3J4 and Y1Y2 respectively; (2.4) Add the features extracted in step (1.7) to the ratio vector mapping function to enhance the anti-interference and anti-attack capabilities of the extracted features.

4. A computer vision-based mental illness identification method as claimed in claim 1, characterized in that: The step 3) of constructing a mental illness classification model specifically includes: (3.1) A triple facial feature stream shallow neural network is designed for the classification of mental illness. For a long mental illness consultation video, it is first divided into short video clips of equal length according to the dialogue or the offset of the landmarks, and the redundant video clips are discarded. Three groups of feature vectors are extracted from the divided short video clips in time sequence, and spliced ​​according to the facial structure to obtain the position feature vector map, distance feature vector map and dynamic feature vector map; (3.2) Input the three sets of feature vectors into the triple facial feature stream shallow neural network. At the beginning of the model, extract the features, perform a convolution and a maximum pooling to maintain the independence of each set of features, then concatenate the high-dimensional features, perform a maximum pooling again, and finally connect to the fully connected layer to output a score value; (3.3) In the design of the triple facial feature stream shallow neural network, the classification task of a single long video is transformed into a regression task of multiple short videos. Each short video clip needs to be pseudo-labeled through the Heaviside step function for training to obtain a weight model.

5. The method for identifying mental illness based on computer vision as claimed in claim 1, characterized in that: Step 4) of identifying the patient's mental illness category specifically includes: (4.1) Select the patient consultation video to be diagnosed, extract the feature point data, cut it according to the facial landmark point offset, and convert the data into short time series unit data; (4.2) Load the triple facial feature stream shallow neural network model saved in step 3, input the short time sequence unit data obtained in (4.1), and obtain the model output scores of multiple short time sequence units; (4.3) Aggregate the scores of the multiple short time sequence units obtained in step 3 to obtain the overall score; (4.4) Based on the overall score obtained in (4.3), output the patient’s mental illness category.

6. A computer vision-based mental illness identification device, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement a computer vision-based mental illness identification method as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a computer vision-based mental illness identification method as described in any one of claims 1 to 5 is implemented.

Images