A mental illness recognition system based on visual sensor collected optical flow features
By using an optical flow feature recognition system based on visual sensors, the problem of time-consuming and costly mental illness detection has been solved, achieving efficient and accurate mental illness screening, simplifying the diagnostic process and reducing costs.
Patent Information
- Application Number
- CN202310571093.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-17
- Publication Date
- 2025-12-16
- Estimated Expiration
- 2043-05-17
AI Technical Summary
Existing technologies for mental illness detection are time-consuming, costly, and lack non-invasive testing methods. Furthermore, facial visual feature recognition methods struggle to capture subtle facial changes, making screening difficult.
A mental illness identification system based on optical flow features acquired by a visual sensor is adopted. Through preprocessing, facial stress unit extraction, optical flow change feature construction and classification model, mental illness is identified. This includes facial ROI region extraction, optical flow calculation and feature stitching, combining HSV color space and pseudo-label function to generate labels, and using a temporal displacement module for classification.
It achieves highly accurate and efficient detection of mental illnesses, simplifies the diagnostic process, reduces costs, and provides a convenient non-invasive testing method.
Smart Images
Figure CN116612515B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of machine vision technology and discloses a mental illness identification system based on optical flow characteristics collected by a visual sensor. Background Technology
[0002] With the fast pace of modern life, people experience pressure from all sides. The accumulation of stress leads to various mental health problems, eventually developing into mental illnesses. These mental illnesses harm people's physical and mental well-being, and in severe cases, can even lead patients to commit acts that threaten their own or others' lives. A recent report from the World Health Organization shows that by the end of 2021, approximately 13% of the global population had varying degrees of mental illness, with annual costs for treatment and prevention reaching trillions of dollars, placing a huge burden and impact on individuals, families, and society.
[0003] The detection and identification of mental illness has always been a complex and time-consuming task. Determining whether someone is suffering from a mental illness requires not only professional knowledge and a basic understanding of their personal information, but also observation of the target population over a specific timeframe for clinical diagnosis. Many people with mild mental illness are often unaware of their abnormal mental state and do not proactively disclose their condition or seek treatment, leading to significant practical difficulties in mental illness screening.
[0004] Visual sensors can accurately acquire information such as facial expressions, gestures, postures, and head movements. Studies have shown that facial expressions account for 55% of emotional information, while vocal features and semantic features account for 38% and 7%, respectively. Therefore, using visual sensors to collect facial changes for mental illness identification is a very effective method.
[0005] Beyond data collection, visual sensor-based methods for identifying mental illnesses offer a cost-effective, convenient, and non-invasive approach to assisted screening. Compared to traditional clinical interviews and questionnaires, visual sensors can directly collect data on patients' daily behaviors and expressions without human intervention, thus providing a more objective reflection of their true condition.
[0006] Furthermore, methods applying overall facial features are very rare in mental illness classification models. Using local facial information easily leads to the loss of a large number of features. While facial landmarks maintain the topological structure of the face, the features they represent are relatively macroscopic and difficult to capture subtle facial changes. In addition, manually constructing features can easily result in the loss of some deep features beyond clinical cognition. This invention constructs a mental illness identification method based on facial visual features by acquiring optical flow features using a visual sensor. Summary of the Invention
[0007] This invention aims to overcome the shortcomings of existing domestic and international methods for identifying mental illnesses based on facial visual features. Starting from visual sensors and facial optical flow features, it proposes a new mental illness identification system based on optical flow features acquired by visual sensors.
[0008] This invention studies the richness of features in different facial regions, identifies three regions of interest for optical flow feature extraction, and can obtain local features with rich expression changes. It can also combine local features to maintain changes in facial topology without losing global feature information. Using patient consultation videos, a highly accurate and efficient method for detecting mental illnesses is realized.
[0009] The mental illness identification system based on optical flow features acquired by a visual sensor, as described in this invention, includes a mental illness expert consultation video data preprocessing module, a facial stress unit extraction module, an optical flow change feature unit construction module, a classification model construction module, and a patient's mental illness category identification module, connected sequentially. The mental illness expert consultation video data preprocessing module feeds patient facial images to the facial stress unit extraction module. The facial stress unit extraction module feeds the optical flow calculation method and the patient's facial image sequence to the optical flow change feature unit construction module. The optical flow change feature unit construction module feeds the optical flow change feature units to the classification model construction module and the patient's mental illness category identification module, respectively. The classification model construction module feeds the classification model to the patient's mental illness category identification module.
[0010] The video data preprocessing module for mental illness expert consultations includes:
[0011] (1.1) Organize the consultation videos of mental illness experts collected using visual sensors, and extract the patient's facial images from the consultation videos frame by frame using the HOG feature description operator;
[0012] (1.2) Align the patient’s facial images using affine transformations and unify the size of the patient’s facial images to 128×128 pixels;
[0013] (1.3) Clean the patient's facial images using the OpenFace toolbox. Define the confidence level as 0.98. Delete the patient's facial images with a confidence level lower than 0.98 and retain the patient's facial images with a confidence level greater than or equal to 0.98.
[0014] The facial stress unit extraction module specifically includes:
[0015] (2.1) Select facial images without expression changes from the patient's facial images as reference frames;
[0016] (2.2) Calculate the optical flow for each frame of the patient's face image and the reference frame. The horizontal and vertical components of the optical flow are calculated using the TV-L1 optical flow method.
[0017] (2.3) The optical flow obtained from each frame of the patient's facial image is further processed to calculate the optical strain, which is defined as shown in (2-1). Wherein, ∈ xx and ∈ yy For the tangent strain component, ∈ xy and ∈ yx Let ∈ be the shear strain component, and ∈ be the optical strain.
[0018]
[0019] (2.4) Define three rectangular Regions of Interest (ROIs) for the patient's facial image: the left eyebrow region, the right eyebrow region, and the mouth region. Use facial landmarks to distinguish these three regions and extract them. When extracting, extend each region by 12 pixels in four directions to ensure that no information is lost.
[0020] (2.5) Adjust the size of the left eyebrow area to 27×27 pixels, the right eyebrow area to 27×27 pixels, and the mouth area to 27×52 pixels;
[0021] (2.6) The left eyebrow region of the left eye, the right eyebrow region of the right eye, and the mouth region are spliced together according to the facial topology to obtain a region of interest map with a size of 54×54;
[0022] (2.7) Based on the positional relationship of the region of interest map, obtain the optical strain of the corresponding region from the original patient facial image;
[0023] (2.8) Calculate the optical strain of the region of interest in each frame of the patient's face image according to the method in (2.7);
[0024] (2.9) Statistically measure the optical strain of each frame to obtain an optical strain set, denoise it by Savitzky-Golay filtering, and use the isolated forest algorithm to detect outliers.
[0025] (2.10) Perform peak detection on the optical strain set obtained in (2.9), count the duration of all peaks, and obtain the set of peak times;
[0026] (2.11) Perform K-Means clustering on the peak time set obtained in (2.10) to obtain a cluster center, which serves as the unit length for dividing patient facial images;
[0027] (2.12) Divide the patient's facial images according to the unit length obtained in (2.11) to obtain multiple patient facial image sequences of a specified length, which are used as the training set;
[0028] The optical flow variation feature unit construction module specifically includes:
[0029] (3.1) The optical flow calculation method in (2.2) is used to process the patient face image sequence obtained in (2.12) to obtain multiple optical flows corresponding to the patient face image sequence. The horizontal component, vertical component and optical strain matrices of the optical flow are stacked to obtain an optical flow feature map of size 54×54×3.
[0030] (3.2) The optical flow feature map obtained in (3.1) is normalized using the HSV color space, and then stitched together into a video at a rate of 16 frames per second, which is a single optical flow variation feature unit.
[0031] (3.3) For the optical flow variation feature units obtained in section (3.2), the corresponding labels are generated using the pseudo-label function, which is shown in formula (3-1);
[0032] g(label) = label (3-1)
[0033] The classification model building module specifically includes:
[0034] (4.1) A classification model is constructed based on the temporal displacement module. The input of the model is the optical flow change feature unit in (3.3). The input optical flow change feature unit is divided into 16 segments of equal size, and PyAV is used as the decoder.
[0035] (4.2) The 16 segments sampled in (4.1) are sent to the optical flow extraction head for feature extraction. The optical flow extraction head performs split extraction on the input image, performs one convolution and one max pooling to maintain the independence of each set of features, then concatenates the high-dimensional features, performs another max pooling, and outputs to the temporal shift module.
[0036] (4.3) The temporal displacement module performs displacement in the time channel, integrates the temporal and spatial features of the input segment, performs training and learning, and saves the classification model;
[0037] The patient's mental illness category identification module specifically includes:
[0038] (5.1) Select the patient's consultation video to be diagnosed and cut it into 8-second segments. Then, convert the video into optical flow change feature units according to the methods in (3.1) and (3.2).
[0039] (5.2) Load the classification model saved in step 4, input the optical flow variation feature unit obtained in (5.1), and obtain the output scores of multiple optical flow variation feature unit models;
[0040] (5.3) The scores of multiple optical flow variation feature units obtained in (5.2) are aggregated to obtain the overall score;
[0041] (5.4) Based on the overall score obtained in (5.3), output the patient's mental illness category.
[0042] This invention enables the classification and identification of samples from mentally ill and normal individuals using existing high-resolution user facial videos. Experimental results show that this invention can effectively screen patients with mental illnesses by extracting optical flow features from data collected by a visual sensor.
[0043] The advantages of this invention are: the method is easy to operate, has a shallow model layer count, short preprocessing time, short inference time, and fast evaluation speed. Currently, the diagnosis of mental illness requires visits to specialized medical institutions, which involves long diagnostic cycles, cumbersome procedures, and high costs. Therefore, this invention can serve as an auxiliary means for mental illness detection. Users only need to record a consultation video using a visual sensor to obtain results through this invention, providing fast feedback and assisting in the diagnosis of mental illness. Attached Figure Description
[0044] Figure 1 This is a schematic diagram of a facial region of interest according to the present invention.
[0045] Figure 2 This is a schematic diagram of the optical flow characteristics of the present invention.
[0046] Figure 3 This is a schematic diagram of the classification model structure of the present invention.
[0047] Figure 4 This is a system structure diagram of the present invention. Detailed Implementation
[0048] The invention will be further described below with reference to the accompanying drawings.
[0049] Reference Figures 1-4 :
[0050] A mental illness identification system based on optical flow features acquired by a visual sensor includes, in sequence, a preprocessing module for video data from expert consultations, a facial stress unit extraction module, an optical flow change feature unit construction module, a classification model construction module, and a patient's mental illness category identification module. The expert consultation video data preprocessing module feeds patient facial images to the facial stress unit extraction module. The facial stress unit extraction module feeds the optical flow calculation method and the patient's facial image sequence to the optical flow change feature unit construction module. The optical flow change feature unit construction module feeds the optical flow change feature units to both the classification model construction module and the patient's mental illness category identification module. The classification model construction module then feeds the classification model to the patient's mental illness category identification module.
[0051] The video data preprocessing module for mental illness expert consultations includes:
[0052] (1.1) Organize the consultation videos of mental illness experts collected using visual sensors, and extract the patient's facial images from the consultation videos frame by frame using the HOG feature description operator;
[0053] (1.2) Align the patient’s facial images using affine transformations and unify the size of the patient’s facial images to 128×128 pixels;
[0054] (1.3) Clean the patient's facial images using the OpenFace toolbox. Define the confidence level as 0.98. Delete the patient's facial images with a confidence level lower than 0.98 and retain the patient's facial images with a confidence level greater than or equal to 0.98.
[0055] The facial stress unit extraction module specifically includes:
[0056] (2.1) Select facial images without expression changes from the patient's facial images as reference frames;
[0057] (2.2) Calculate the optical flow for each frame of the patient's face image and the reference frame. The horizontal and vertical components of the optical flow are calculated using the TV-L1 optical flow method.
[0058] (2.3) The optical flow obtained from each frame of the patient's facial image is further processed to calculate the optical strain, which is defined as shown in (2-1). Wherein, ∈ xx and ∈ yy For the tangent strain component, ∈ xy and ∈ yx Let ∈ be the shear strain component, and ∈ be the optical strain.
[0059]
[0060] (2.4) Define three rectangular Regions of Interest (ROIs) for the patient's facial image: the left eyebrow region, the right eyebrow region, and the mouth region. Use facial landmarks to distinguish these three regions and extract them. When extracting, extend each region by 12 pixels in four directions to ensure that no information is lost.
[0061] (2.5) Adjust the size of the left eyebrow area to 27×27 pixels, the right eyebrow area to 27×27 pixels, and the mouth area to 27×52 pixels;
[0062] (2.6) The left eyebrow region of the left eye, the right eyebrow region of the right eye, and the mouth region are spliced together according to the facial topology to obtain a region of interest map with a size of 54×54;
[0063] (2.7) Based on the positional relationship of the region of interest map, obtain the optical strain of the corresponding region from the original patient facial image;
[0064] (2.8) Calculate the optical strain of the region of interest in each frame of the patient's face image according to the method in (2.7);
[0065] (2.9) Statistically measure the optical strain of each frame to obtain an optical strain set, denoise it by Savitzky-Golay filtering, and use the isolated forest algorithm to detect outliers.
[0066] (2.10) Perform peak detection on the optical strain set obtained in (2.9), count the duration of all peaks, and obtain the set of peak times;
[0067] (2.11) Perform K-Means clustering on the peak time set obtained in (2.10) to obtain a cluster center, which serves as the unit length for dividing patient facial images;
[0068] (2.12) Divide the patient's facial images according to the unit length obtained in (2.11) to obtain multiple patient facial image sequences of a specified length, which are used as the training set;
[0069] The optical flow variation feature unit construction module specifically includes:
[0070] (3.1) The optical flow calculation method in (2.2) is used to process the patient face image sequence obtained in (2.12) to obtain multiple optical flows corresponding to the patient face image sequence. The horizontal component, vertical component and optical strain matrices of the optical flow are stacked to obtain an optical flow feature map of size 54×54×3.
[0071] (3.2) The optical flow feature map obtained in (3.1) is normalized using the HSV color space, and then stitched together into a video at a rate of 16 frames per second, which is a single optical flow variation feature unit.
[0072] (3.3) For the optical flow variation feature units obtained in section (3.2), the corresponding labels are generated using the pseudo-label function, which is shown in formula (3-1);
[0073] g(label) = label (3-1)
[0074] The classification model building module specifically includes:
[0075] (4.1) A classification model is constructed based on the temporal displacement module. The input of the model is the optical flow change feature unit in (3.3). The input optical flow change feature unit is divided into 16 segments of equal size, and PyAV is used as the decoder.
[0076] (4.2) The 16 segments sampled in (4.1) are sent to the optical flow extraction head for feature extraction. The optical flow extraction head performs split extraction on the input image, performs one convolution and one max pooling to maintain the independence of each set of features, then concatenates the high-dimensional features, performs another max pooling, and outputs to the temporal shift module.
[0077] (4.3) The temporal displacement module performs displacement in the time channel, integrates the temporal and spatial features of the input segment, performs training and learning, and saves the classification model;
[0078] The patient's mental illness category identification module specifically includes:
[0079] (5.1) Select the patient's consultation video to be diagnosed and cut it into 8-second segments. Then, convert the video into optical flow change feature units according to the methods in (3.1) and (3.2).
[0080] (5.2) Load the classification model saved in step 4, input the optical flow variation feature unit obtained in (5.1), and obtain the output scores of multiple optical flow variation feature unit models;
[0081] (5.3) The scores of multiple optical flow variation feature units obtained in (5.2) are aggregated to obtain the overall score;
[0082] (5.4) Based on the overall score obtained in (5.3), output the patient's mental illness category.
Claims
1. A mental illness identification system based on optical flow features acquired by a visual sensor, characterized in that: The system comprises, in sequence, a preprocessing module for video data from consultations with mental illness experts, a facial stress unit extraction module, an optical flow change feature unit construction module, a classification model construction module, and a patient's mental illness category identification module. The preprocessing module feeds patient facial images to the facial stress unit extraction module. The facial stress unit extraction module feeds the optical flow calculation method and the patient's facial image sequence to the optical flow change feature unit construction module. The optical flow change feature unit construction module feeds the optical flow change feature units to both the classification model construction module and the patient's mental illness category identification module. Finally, the classification model construction module feeds the classification model to the patient's mental illness category identification module. The video data preprocessing module for mental illness expert consultations includes: (1.1) Organize the consultation videos of mental illness experts collected using visual sensors, and extract the patient's facial images from the consultation videos frame by frame using the HOG feature description operator; (1.2) Align the patient’s facial images using affine transformations and unify the size of the patient’s facial images to 128×128 pixels; (1.3) Clean the patient's facial images using the OpenFace toolbox. Define the confidence level as 0.
98. Delete the patient's facial images with a confidence level lower than 0.98 and retain the patient's facial images with a confidence level greater than or equal to 0.
98. Step 2), Facial Stress Unit Extraction Module, specifically includes: (2.1) Select facial images without expression changes from the patient's facial images as reference frames; (2.2) Calculate the optical flow for each frame of the patient's face image and the reference frame. The horizontal and vertical components of the optical flow are calculated using the TV-L1 optical flow method. (2.3) The optical flow obtained from each frame of the patient's facial image is further processed to obtain the optical strain, which is defined as shown in (2-1); where, ∈ xx and ∈ yy For the tangent strain component, ∈ xy and ∈ yx Let ∈ be the shear strain component, and ∈ be the optical strain. (2.4) Define three rectangular ROI regions for the patient's facial image: the left eyebrow region, the right eyebrow region, and the mouth region. Use facial landmarks to distinguish these three regions and extract them. When extracting, extend each region by 12 pixels in four directions to ensure that no information is lost. (2.5) Adjust the size of the left eyebrow area to 27×27 pixels, the right eyebrow area to 27×27 pixels, and the mouth area to 27×52 pixels; (2.6) The left eyebrow region of the left eye, the right eyebrow region of the right eye, and the mouth region are spliced together according to the facial topology to obtain a region of interest map with a size of 54×54; (2.7) Based on the positional relationship of the region of interest map, obtain the optical strain of the corresponding region from the original patient facial image; (2.8) Calculate the optical strain of the region of interest in each frame of the patient's face image according to the method in (2.7); (2.9) Statistically measure the optical strain of each frame to obtain an optical strain set, denoise it by Savitzky-Golay filtering, and use the isolated forest algorithm to detect outliers. (2.10) Perform peak detection on the optical strain set obtained in (2.9), count the duration of all peaks, and obtain the set of peak times; (2.11) Perform K-Means clustering on the peak time set obtained in (2.10) to obtain a cluster center, which serves as the unit length for dividing patient facial images; (2.12) Divide the patient's facial images according to the unit length obtained in (2.11) to obtain multiple patient facial image sequences of a specified length, which are used as the training set; Step 3) Construction module for optical flow variation feature units, specifically including: (3.1) The optical flow calculation method in (2.2) is used to process the patient face image sequence obtained in (2.12) to obtain multiple optical flows corresponding to the patient face image sequence. The horizontal component, vertical component and optical strain matrices of the optical flow are stacked to obtain an optical flow feature map of size 54×54×3. (3.2) The optical flow feature map obtained in (3.1) is normalized using the HSV color space, and then stitched together into a video at a rate of 16 frames per second, which is a single optical flow variation feature unit. (3.3) For the optical flow variation feature units obtained in section (3.2), the corresponding labels are generated using the pseudo-label function, which is shown in formula (3-1); g(label) = label (3-1) The classification model building module specifically includes: (4.1) A classification model is constructed based on the temporal displacement module. The input of the model is the optical flow change feature unit in (3.3). The input optical flow change feature unit is divided into 16 segments of equal size, and PyAV is used as the decoder. (4.2) The 16 segments sampled in (4.1) are sent to the optical flow extraction head for feature extraction. The optical flow extraction head performs split extraction on the input image, performs one convolution and one max pooling to maintain the independence of each set of features, then concatenates the high-dimensional features, performs another max pooling, and outputs to the temporal shift module. (4.3) The temporal displacement module performs displacement in the time channel, integrates the temporal and spatial features of the input segment, performs training and learning, and saves the classification model; The patient's mental illness category identification module specifically includes: (5.1) Select the patient's consultation video to be diagnosed and cut it into 8-second segments. Then, convert the video into optical flow change feature units according to the methods in (3.1) and (3.2). (5.2) Load the classification model saved in step 4, input the optical flow variation feature unit obtained in (5.1), and obtain the output scores of multiple optical flow variation feature unit models; (5.3) The scores of multiple optical flow variation feature units obtained in (5.2) are aggregated to obtain the overall score; (5.4) Based on the overall score obtained in (5.3), output the patient's mental illness category.