Mental health emotion deep interaction method, system and equipment based on large model

Abstract

The invention discloses a psychological health emotion deep interaction method, device and equipment based on a large model. The method comprises the following steps: collecting face video data and medical text information of a psychological health patient; acquiring a medical text feature vector and a face video feature vector; mapping the medical text feature vector and the face video feature vector to the same semantic space for deep alignment, and constructing a cross-modal retrieval model; historical medical record data similar to the to-be-tested person is retrieved through the cross-modal retrieval model; generating a face dynamic broken line graph according to the face video feature vector of the to-be-tested person; generating an interpretable diagnosis report according to the historical medical records similar to the to-be-tested person retrieved by the cross-modal retrieval model and the medical data of the historical medical records; and performing three-dimensional face reconstruction according to the face video data of the to-be-detected person, comparing the three-dimensional face with the three-dimensional facial makeup library, and outputting the similarity with the typical condition facial makeup. According to the method, end-to-end automatic generation from original data to multi-dimensional and explainable auxiliary diagnosis results is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of healthcare informatics, and particularly relates to a mental health emotion deep interaction method, system and device based on a large model. BACKGROUND

[0002] With the increase of social pressure and the acceleration of life rhythm, the incidence of mental diseases is on the rise, among which unipolar affective disorder and bipolar affective disorder are relatively common and seriously affect the quality of life of patients. Early screening, accurate diagnosis and disease prediction are crucial for the treatment and management of mental diseases, however, the traditional diagnosis method mainly relies on the subjective evaluation of the doctor on the patient's symptoms and the scale score, and there are problems such as the diagnosis accuracy being greatly influenced by the doctor's experience and being difficult to capture the subtle and dynamic emotional changes of the patient.

[0003] At present, the mental disease auxiliary diagnosis model based on deep learning can significantly improve the diagnosis accuracy and comprehensiveness through multi-modal data fusion analysis, use convolutional neural network (CNN) to extract brain structure and functional connection features (such as hippocampal atrophy quantification of Alzheimer's disease) from MRI, fMRI and other images, use recurrent neural network (RNN) or Transformer to process EEG, eye tracking and other physiological signals to capture abnormal patterns, and use natural language processing (NLP) model (such as BERT) to analyze structured information (such as depression attack frequency) in electronic medical records, and finally integrate cross-modal features through splicing, weighted summation or attention mechanism to form a joint vector; by constructing independent sub-models of images, physiological signals and texts, the diagnosis probability is output respectively.

[0004] In recent years, with the development of computer technology and artificial intelligence, multi-modal fusion has become a research hotspot. In addition to traditional image, physiological and clinical text data, the integration of facial video and medical text is particularly key - the subtle changes of facial action units (AUs) can reflect emotional state (such as AU4 continuous activation indicating anxiety), and medical text contains core information such as medical history and scale score, and the integration of the two can more comprehensively excavate disease characteristics, how to efficiently realize this cross-modal interaction learning has become a key challenge to improve the reliability of diagnosis.

[0005] The existing related technologies have made certain progress in multi-modal data processing and fusion, but still have some deficiencies. For example, some technologies only focus on the analysis of single modal data, ignoring the correlation and complementarity between different modal data; some methods fail to fully consider the regularity of facial motion units and key points changing over time when processing facial video data, resulting in insufficient modeling capability for micro-expression dynamic changes; lack of multi-dimensional data interpretation and visualization: the technical solution mainly focuses on the fusion and classification prediction of multi-modal data, lacking multi-dimensional interpretation and visual display of data. In clinical auxiliary diagnosis, doctors need intuitive and visual information to assist in judgment, but the technical solution cannot provide intuitive visual results such as facial dynamic fold line chart, three-dimensional face spectrum matching, etc., which is not conducive to doctors to make quick and accurate diagnosis decisions. In addition, the accuracy and interpretability of the existing technology in cross-modal alignment and retrieval need to be improved, which is difficult to meet the actual needs of clinical auxiliary diagnosis.

[0006] In summary, the existing related technical solutions provide a certain basis for the auxiliary diagnosis of mental diseases in multi-modal data processing and fusion, but still have many deficiencies. SUMMARY

[0007] The present application provides a mental health emotion deep interaction method based on large model, which realizes deep fusion and effective analysis of multi-modal data, and improves the accuracy, reliability and interpretability of auxiliary diagnosis of mental diseases.

[0008] The technical solution of the present application is as follows: A mental health emotion deep interaction method based on large model, comprising the following steps: (1) collecting facial video data and medical text information of mental health disease patients and preprocessing; (2) encoding the medical text information to obtain a medical text feature vector; extracting facial motion unit time series feature vectors and 68 facial key point time series feature vectors from the facial video data, and performing weighted fusion to obtain a facial video feature vector; (3) mapping the medical text feature vector and the facial video feature vector to the same semantic space for deep alignment, and constructing a cross-modal retrieval model; retrieving similar historical medical records and medical data of the to-be-tested person from the known medical record database through the cross-modal retrieval model; (4) generating a facial dynamic fold line chart from the time series feature vector extracted from the facial motion unit intensity sequence of the to-be-tested person; generating an interpretable diagnosis report according to the similar historical medical records and medical data of the to-be-tested person retrieved by the cross-modal retrieval model; performing three-dimensional face reconstruction on the facial video data of the to-be-tested person and comparing with a three-dimensional face spectrum library to output the similarity with a typical disease spectrum.

[0009] Preferably, in step (1), the medical text information includes somatic disease diagnosis (scoring criteria), somatic disease medication (scoring criteria), whether there are negative ideas, whether there are negative behaviors, whether there is a history of emotional disorder in the family, whether there is a combined mental illness, the current disease and combined disease and medication (including medication name), PHQ-9 scale score, Hamilton depression scale score, Hamilton anxiety scale score, Yang's mania rating scale score.

[0010] Preferably, in step (1), the collected data includes clinical data and data on social platforms.

[0011] The mental health disease patient includes a unipolar affective disorder patient and a bipolar affective disorder patient.

[0012] Preferably, step (2) includes: (2-1) using a text encoder to convert each text item in the medical text information into a semantic vector, using a multi-head self-attention mechanism to weight and fuse the semantic vectors of all text items to obtain a medical text feature vector ; (2-2) extracting the activation intensity value of each frame of facial motion unit from the facial video data, constructing a facial motion unit activation intensity sequence and inputting it into an LSTM network to obtain a facial motion unit time sequence feature vector ; using a key point tracking algorithm based on deep learning to identify a 68 facial key point coordinate sequence in the facial video data, and encoding the 68 facial key point coordinate sequence into a 68 facial key point time sequence feature vector through a 3D-CNN network ; (2-3) fusing and through an attention weighting mechanism to obtain a facial video feature vector .

[0013] Preferably, step (3) includes: (3-1) mapping the medical text feature vector and the facial video feature vector to the same semantic space; (3-2) using a cosine similarity contrast loss function to train the model, so that the facial video feature vector of the patient is as close as possible to its corresponding medical text feature vector , and the facial video feature vector of the patient is as far away as possible from the medical text feature vectors of other patients, and a cross-modal retrieval model is obtained after training is completed; (3-3) Extract the facial video feature vectors of known cases and medical text feature vectors Construct a database of known medical records; (3-4) Use a cross-modal retrieval model to retrieve historical medical records and medical data similar to the subject from a known medical record database.

[0014] The cross-modal retrieval model supports two retrieval modes: image or video search. When a new patient's facial video data is input, the cross-modal retrieval model will search based on their facial video feature vector. Calculate its feature vector with all medical texts in a known medical record database. The similarity search returns the most similar medical text records; for text-based image or video search, input a text description (e.g., "Looking for patients with manic symptoms"), and the cross-modal retrieval model will search based on its medical text feature vectors. Calculate it with all facial video feature vectors Based on the similarity, return the few facial videos with the highest similarity.

[0015] Preferably, step (4) includes: (4-1) Extract the activation intensity value of facial motion units from each frame of facial video data, construct the activation intensity sequence of facial motion units, input it into the LSTM network, and cluster the output of the LSTM network at each time step according to the disease type. With the X-axis as time and the Y-axis as the average activation intensity, draw a dynamic line graph for each facial motion unit of each disease type. (4-2) Input the historical medical records and medical data similar to the subject retrieved by the cross-modal retrieval model into the fine-tuned large language model to generate an explanatory report in natural language form; (4-3) Perform three-dimensional face reconstruction based on the facial video data of the subject, and use 3D-CNN to encode the three-dimensional face reconstruction results, and calculate the similarity between the encoded results and typical disease face in the three-dimensional face database; (4-4) Visualize the results and similarity of the 3D face reconstruction.

[0016] Furthermore, the method for constructing the 3D face database includes: collecting facial video data of patients with different types of clinically diagnosed mental illnesses and healthy individuals; using a 3D face reconstruction algorithm to generate a corresponding 3D face model for each individual; subsequently, grouping the models according to clinical diagnostic labels, and generating typical condition faces representing various typical conditions by calculating the spatial geometric mean of the 3D models within each group.

[0017] Based on the same inventive concept, the present invention also provides a large-scale model-based deep interactive device for mental health and emotions, comprising the following units: The data acquisition unit collects and preprocesses facial video data and medical text information from patients with mental health disorders. The multimodal feature extraction unit encodes medical text information to obtain medical text feature vectors; it extracts temporal feature vectors of facial motion units and 68 facial key points from facial video data, and performs weighted fusion to obtain facial video feature vectors. The cross-modal alignment and retrieval module maps medical text feature vectors and facial video feature vectors to the same semantic space for deep alignment, constructing a cross-modal retrieval model; and retrieves historical medical records and medical data similar to the subject from a known medical record database through the cross-modal retrieval model. The results generation module generates a dynamic facial polygonal map based on the temporal feature vector extracted from the facial motion unit intensity sequence of the subject; it generates an interpretable diagnostic report based on the historical medical records and medical data similar to the subject retrieved by the cross-modal retrieval model; and it performs three-dimensional face reconstruction based on the subject's facial video data and compares it with a three-dimensional face database to output the similarity with typical disease face patterns.

[0018] Based on the same inventive concept, the present invention also provides a large-model-based psychological and emotional deep interaction device, including a memory and a processor. The memory stores a computer program that can be executed on the processor. When the processor executes the computer program, it implements the large-model-based psychological and emotional deep interaction method.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention enables end-to-end automatic generation of multi-dimensional, interpretable auxiliary diagnostic results from raw facial video data and medical text information. It can provide intuitive visualization results such as dynamic facial line graphs and 3D face matching, which can assist doctors in making diagnostic decisions quickly and accurately. In addition, through cross-modal alignment and retrieval, it can generate auxiliary diagnostic reports with high accuracy and interpretability, which can meet the actual needs of clinical auxiliary diagnosis. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a deep interaction method for mental health and emotions based on a large model. Figure 2 This is a flowchart illustrating the process of feature extraction, cross-modal alignment, and retrieval. Figure 3 A flowchart illustrating the process of generating results for the three branches; Figure 4 This is a flowchart illustrating the method for generating and matching 3D faces. Detailed Implementation

[0021] The application will be described in further detail below in conjunction with the accompanying drawings and examples. It should be noted that the examples described below are intended to facilitate an understanding of the application and do not in any way limit the same.

[0022] As shown in Figure 1 The large model-based mental health emotion deep interaction method includes the following steps: 1. Dataset: clinical modal data and social modal data; (S100) 1.1 Data collection part Clinical facial video data (video data 1), facial video data of patients with unipolar affective disorder and patients with bipolar affective disorder provided by the hospital.

[0023] Structured medical text information (text information 1), 11 items of basic information corresponding to the patients in video data 1 provided by the hospital, respectively: somatic disease diagnosis (scoring standard), somatic disease medication (scoring standard), whether there are negative ideas, whether there are negative behaviors, whether there is a history of emotional disorder in the family, whether there is a combined mental illness attack, current disease and combined disease and medication (including medication name), PHQ-9 scale score, Hamilton depression scale score, Hamilton anxiety scale score, Yang's mania rating scale score.

[0024] Social modal video data (video data 2), video data of patients with unipolar and bipolar affective disorder is crawled from social software such as Xiaohongshu, TikTok, etc. through crawler.

[0025] Social modal text data (text information 2): information crawled from related user scripts and comment areas is arranged into text, and advertisements, non-medical related content are removed to improve the quality and availability of data, so that the data is more in line with the needs of medical research or analysis.

[0026] 1.2 Data preprocessing part: (S101) (1) Video data preprocessing: In order to solve the problems of complex picture, uneven illumination, high noise interference and different time length of social platform video, system preprocessing is needed.

[0027] First, video screening and cleaning: retain emotional related segments for keywords and labels, use OpenFace to complete face detection and key point recognition, exclude multi-person, occlusion or non-frontal face video, and unify time length and frame rate to match the clinical data standard.

[0028] Then, extract the facial action unit intensity sequence and 68 facial key point coordinates, and convert them into a standardized time sequence feature matrix. Automatically blur or replace the background to ensure that only the facial region and time sequence data are retained.

[0029] Finally, the structured data containing facial action unit matrix, key point coordinate matrix and anonymous label is outputted and saved in.csv format. The preprocessing procedure aligns the social video in time dimension and facial feature expression with the clinical video.

[0030] In the present application, an efficient video preprocessing procedure is designed and implemented to extract standardized single-person facial data from large-scale raw video data (including video data 1 and video data 2) for tasks such as psychological state analysis. Specifically, the time-series motion features of facial action units (such as AU4 frown and AU12 mouth corner up) are extracted, and 11 items of structured medical text corresponding to the patient (including: body disease diagnosis (score), body disease medication (score), presence or absence of negative ideas, presence or absence of negative behavior, family history of emotional disorder, presence or absence of combined mental disease, current disease and combined disease and medication (including medication name), PHQ-9 scale score, Hamilton depression scale score, Hamilton anxiety scale score, and Yang's mania rating scale score) are labeled.

[0031] The procedure first uses the MediaPipe face detection module to strictly screen the video frames, retaining only frames with a confidence level higher than 0.9 and detecting a single person's frontal face, ensuring data quality. Subsequently, all valid frames are standardized segmented by 20 seconds (25FPS x 20 seconds = 500 frames), and video segments less than one segment will be discarded to ensure time consistency. To control the amount of output data, a segment number limit parameter (NPART, default maximum 25 segments) is introduced, and frames exceeding the threshold will be ignored, which helps to achieve sample balance in multi-participant scenarios.

[0032] In terms of output form, the system supports dual-mode output: segmented video (mp4 format) and frame-by-frame picture (jpg format), which is convenient for various subsequent analysis tasks. The present application also designs an intelligent ID grouping mechanism that automatically identifies and classifies participant IDs based on file names, supports multiple video formats (such as.mp4,.avi,.ts), and automatically skips non-standardly named files. In addition, to improve processing efficiency, the system uses a multi-thread parallel processing architecture (up to 4 threads), ensuring that each thread independently creates a MediaPipe detection instance, effectively avoiding resource competition. The preprocessing procedure is particularly suitable for the needs of unified and controllable extraction of multi-participant video data in psychological experiments.

[0033] (2) Text information preprocessing: Structured medical text: The clinical reports are strictly pre-processed, including text extraction, cleaning and segmentation. The present application uses Google Cloud Plateform and Document AI tools for optical character recognition (OCR) to convert scanned reports into pure text format. To ensure the accuracy and consistency of the text, the text in the report is first extracted, and the header, footer and other non-text content are removed. Then the extracted text is cleaned to remove repeated character features and format errors. Finally, the report is segmented into individual sentences so that the model can analyze sentence by sentence. The pre-training stage uses the RoBERTa model to train on large-scale text data, enabling it to understand and process various natural language inputs.

[0034] The structured medical text information (text information 1) and the social modal text data (text information 2) are pre-processed in the same way as described above.

[0035] 2. Model architecture (S200) The CLIP (Contrastive Language-Image Pre-training) model is a powerful cross-modal pre-training model that aligns text and images through contrastive learning, enhancing spatial correlation. The present application proposes a multi-modal analysis framework for the auxiliary diagnosis of mental diseases, based on the innovative improvement of the CLIP model framework.

[0036] On this basis, the present application fully integrates text and image multi-modal information, and deeply excavates key features related to the diagnosis of mental diseases, especially focusing on facial motion units and the regularity of 68 facial key points changing over time, to improve the accuracy and reliability of auxiliary diagnosis of mental diseases. The improved model mainly includes the following key modules: 2.1 Key components based on the CLIP base model framework Text encoder: The Transformer architecture (a deep learning model architecture based on self-attention mechanism, which can efficiently process sequence data) is used to deeply encode the input text description covering patient self-reports, doctor's diagnosis records, scale descriptions, etc., converting the text into a format that the model can learn, containing key information such as emotions, symptoms, behaviors, etc. A multi-layer attention mechanism (a method that dynamically allocates attention to different parts of the input data at different levels, allowing the model to capture key information more comprehensively and deeply) is used to facilitate the subsequent deep alignment between facial video features and medical text.

[0037] The text encoder is used to encode the structured medical text after data preprocessing, including: First, each text item (e.g., "PHQ-9 score: 15", "Have negative thoughts") is converted into a semantic vector; Then, use multi-head self-attention mechanism to let these text items interact with each other. For example, the model can learn that "Have negative thoughts" and "HAMD score high" are strongly related and give them higher attention; Finally, all weighted text features are aggregated (e.g., by labeled output or averaging) to form a medical text feature vector representing the case Image encoder: Based on the Transformer architecture, the input face image is feature extracted, and the facial expression, facial structure, etc. Information in the image is converted into a high-dimensional image feature vector, which can pay attention to the subtle changes of the face, so that the image information can be compared and integrated with the text information in the semantic space (Semantic space is an abstract concept space that maps semantic information into mathematical representation. In this space, semantically similar content will be relatively close to each other, facilitating semantic-level analysis and calculation).

[0038] Video text alignment module: First, the video encoder extracts frame-level and temporal features from the input video; Video encoder workflow: input video - split into frames - image encoder (backbone network) processes each frame - get frame-level feature sequence - temporal modeling module analyzes the feature sequence - get video features containing spatio-temporal information. At the same time, the text encoder converts each action category into a natural language description and maps it to the same semantic space. Through contrastive learning, the model optimizes the similarity between the video and the corresponding text description, so that multi-modal features of the same semantics are tightly aggregated in the same space, see equation (1): (1); Where, s ( v, t ) represents the cosine similarity between video V and text T; Z v represents the video encoding feature vector, which is obtained by the video encoder extracting frame-level and temporal features from the input video V; Z t represents the text encoding feature vector, which is obtained by the text encoder encoding the input text T.

[0039] The video encoder converts the input video into a video encoding vector Z v representing its visual semantics, and the text encoder converts the action or emotion description into a text encoding vector Z t representing its text semantics. Both are measured by calculating the cosine similarity s ( v, t ) between the video and the text in the semantic space, thereby achieving cross-modal alignment.

[0040] In the prediction and inference module, the model calculates the similarity between the input video and all texts, and selects the category with the highest score as the prediction result.

[0041] 2.2 New key modules such as Figure 2 As shown S201: Temporal Feature Enhancement Module: This module adds features to the image coding section to target the changes in facial motion units and 68 facial key points over time.

[0042] S202: Temporal processing for facial motion units: The detection algorithm is used to identify the state of each facial motion unit from a continuous sequence of facial images. The temporal information module is used to model the change of activation state over time. By learning the temporal feature vector, the ability to model the dynamic changes of micro-expressions is improved.

[0043] A sequence of consecutive facial images is input into an LSTM (Long Short-Term Memory) network to capture the activation changes of facial muscle groups over time, thereby obtaining the temporal feature vector of facial motion units. .

[0044] In the temporal modeling of facial motion units, the update of the LSTM state is shown in formulas (2)-(7): (2); (3); (4); (5); (6); (7); in, Let be the AU intensity at time t, with dimension . d Each dimension corresponds to the activation intensity of a specific facial action unit; for t The hidden state at time -1 stores historical time sequence information; , , These represent the activation values ​​of the forget gate, input gate, and output gate, respectively. Candidate memory states; This represents the current state of the memory cell; This is the current hidden state; each gating unit uses a weight matrix. , , , and bias terms , , 、 Implementing linear mapping and nonlinear regulation; is a sigmoid function; is a hyperbolic tangent function, represents an element-wise multiplication operation.

[0045] S203: 68 facial key point time series processing: using a deep learning-based key point tracking algorithm, identifying 68 key points in a continuous facial image sequence, obtaining different time point position coordinates to form a key point position time series, and using a 3D convolutional neural network (3D-CNN) to encode into a 68 key point time series feature vector .

[0046] The 3D convolution feature extraction part is shown in formula (8): (8); wherein, represents the convolution result of the output feature map at position (x, y, z); is the feature value of the input feature map at the offset position (x+i, y+j, z+k); W(i, j, k) is the weight parameter of the three-dimensional convolution kernel at the corresponding position; b is the bias term.

[0047] By weighted summation in three dimensions (i, j, k), the local weighted aggregation of the input features in the space-time domain is obtained, and the output feature map .

[0048] S204: Time series information fusion: through an attention weighting mechanism, the time series feature vectors of the facial motion unit and the 68 facial key points are fused to obtain a more comprehensive facial time series information representation +(1- ) , wherein is a dynamic weight and is automatically adjusted according to the importance of each modality learned by the model during the training phase.

[0049] The output is taken as the video facial feature vector and is sent to the cross-modal alignment and retrieval module of CLIP for alignment with the text encoding result.

[0050] S205: Cross-modal alignment and retrieval module: the text encoder embeds relevant information and uses a multi-layer attention mechanism to realize deep alignment between facial video features and medical text, supporting cross-modal retrieval of similar cases (cross-modal retrieval is to break the boundaries between different modal data such as text, images, and audio, and to realize cross-form information search. It can search for relevant content in another modality by users through the semantic association between different modalities).

[0051] comprises: (1) mapping the facial video feature vector and the medical text feature vector into the same high-dimensional semantic space, where the distance between vectors represents the similarity of semantics; (2) training the cross-modal alignment and retrieval model using the cosine similarity contrast loss function, so that the distance between the facial video features of the same patient and their corresponding medical text features in the shared space is as close as possible, while the distance between the facial video features of the same patient and the medical text features of other patients is as far apart as possible; achieving deep alignment of facial dynamic patterns and medical text semantics. By continuously optimizing the contrast loss function, the model learns how subtle changes in facial expressions (such as frequent activation of AU4) correspond to descriptions such as "high anxiety scale score" in the text, and thus maps them to similar positions in the semantic space.

[0052] (3) storing the facial video feature vector and the medical text feature vector of known cases in the database and establishing an index; In the inference phase, two retrieval modes are supported: Text search with image or video: input a new patient's facial video into the trained cross-modal alignment and retrieval module, the model will generate its facial video feature vector , then calculate its similarity with all medical text feature vectors in the database, and return the top several medical text records with the highest similarity; Text search with image or video: input a new patient's facial video into the trained cross-modal alignment and retrieval module, the model will generate its facial video feature vector , then calculate its similarity with all medical text feature vectors in the database, and return the top several medical text records with the highest similarity;

[0053] The retrieval results will be directly output as the core basis for generating explainable reports and assisting in diagnostic decision-making. Use the retrieved most similar historical cases to extract their medical information (such as scale scores, diagnosis results) as the key basis for generating the final natural language report. The retrieved similar pathological retrieval results are provided to the second branch of the multi-branch generation module to generate explainable reports.

[0054] 2.3 Multi-branch generation module as shown in Figure 3 (S300) In the model inference stage, the application designs three generation branches to realize multi-dimensional interpretation of data and clinical auxiliary application: S301: First branch: face dynamic fold line chart generation module This module aims to extract the dynamic characteristics of facial action units (AUs, Action Units) changing over time through time series modeling, and realize the quantitative visualization of the correlation between facial expressions and disease. The system first extracts the AU intensity value of each frame from the input video, and constructs its time sequence (for example, the combined frequency of AU6 and AU12 per second). Then, the activation curve of each AU is modeled using LSTM (Long Short-Term Memory, Long Short-Term Memory Network) to capture its dynamic change rule. According to the type of disease (such as unipolar depression, bipolar mania, etc.), the time sequence characteristics of AUs are classified and aggregated, and finally the dynamic fold line chart of AUs changing over time is output. This chart can be used to intuitively compare the expression feature differences between different pathological groups, for example, depressed patients often show high activity of AU4 (frown) and AU15 (mouth corner pull down), while manic patients show frequent activation of AUs such as AU12 (smile) and AU25 (open mouth). This module can provide interpretable intermediate representation for subsequent disease discrimination.

[0055] Including the following steps: The standardized face video segment after preprocessing (S102) is processed using the mature face analysis tool OpenFace for each frame of the video, and the activation intensity of each facial action unit (AU) in each frame is output. The activation intensity is usually a continuous value (for example, 0-5), where 0 indicates no activation, and the larger the value, the higher the muscle contraction intensity.

[0056] For a face video, a T×N matrix will be obtained, where T is the total number of frames (time sequence), and N is the number of detected AUs (N=17 indicates that 17 AUs are detected).

[0057] The T×N-dimensional AU activation intensity matrix described above is modeled using LSTM. Since blinking, speaking and other non-emotional actions, if the fold line chart generated directly using the original AU sequence may have a lot of noise. The role of LSTM is to denoise and smooth, and to capture the internal dynamic change pattern (the frown of depressed patients may be persistent, while the frown of normal people is transient).

[0058] The entire AU sequence is input into an LSTM network, and the time series context information up to the current time is captured. At the same time, the output of LSTM at each time step: (OUTPUT 1) Generate smoothed curves: pass the LSTM's output at each time step through a fully connected layer that maps back to an N-dimensional vector. This vector represents the model's understanding and smoothed version of the current time's AU intensity. The resulting new sequence will be smoother and better reflect the emotional trend than the original sequence.

[0059] (OUTPUT 2) for auxiliary classification: the LSTM's final state can serve as a summary of the AU dynamics for the entire video segment, which can be input into a subsequent classifier (S302 second branch) to aid in diagnosis.

[0060] Finally, we perform: (1) Classification aggregation: Instead of plotting a line for each individual patient, we aim to showcase typical patterns for different types of conditions. Suppose we now have a dataset labeled with condition types, healthy controls, unipolar depression, and bipolar mania. For all patients in the dataset, perform the above steps to obtain their smoothed AU time-series data. Then, group them by condition type. Within each group, average the intensity of the same AU at the same time point (e.g., the average of AU4 intensity at the 5th second for all patients diagnosed with "severe depression"), and calculate the standard deviation.

[0061] (2) Generate dynamic line charts: X-axis is time (unit: seconds or frames); Y-axis is the average activation intensity of AU.

[0062] Draw a curve for each AU of interest (e.g., AU4, AU12, AU15, etc.). Each curve represents the average AU activation pattern for a certain patient group. Use shaded areas to represent standard deviations to reflect intra-group differences. To better demonstrate the dynamic effect: you can use animation to show the entire process of drawing curves over time, enhancing the dynamic observation effect.

[0063] (3) Output and annotation: Finally, output multiple line charts, such as: AU4 (frown) dynamic comparison between unipolar depression group and healthy control group; AU12 (smile) dynamic comparison between bipolar mania group and healthy control group.

[0064] Clinical annotation: Mark key clinical findings on the chart with labels or text, such as: "As shown in the figure, the AU4 (frown) intensity of the depression group is consistently higher than that of the healthy group throughout the observation period", "The AU12 (smile) of the mania group shows higher frequency and intensity of sudden activation".

[0065] S302: Second branch: Explainable report generation (pathological reasoning module) This module constructs a multimodal joint embedding representation based on the AUs features of the input video and social media text keywords (such as "insomnia"), and retrieves similar historical cases from a known medical record database to automatically generate interpretable reports for doctors. The specific process includes: first, calculating the similarity between the input sample and the facial features of existing patients in the database through a cross-modal alignment mechanism, retrieving the most similar sample group; then, returning their structured medical information, such as scores from the PHQ-9 (Personalized Rating Depression Scale), HAMD (Hamilton Depression Scale), HAMA (Hamilton Anxiety Scale), and YMRS (Young's Mania Scale), as well as records of self-harm or other negative behavioral tendencies. Finally, the module calls a finely tuned GPT-3.5 language model to generate an interpretable report in natural language based on the above information, such as: "This patient's AU4 activity level has an 82% similarity to historical cases with PHQ-9 ≥ 15." This module significantly improves the readability and clinical applicability of the model output, avoiding the "black box model" problem.

[0066] The model combines visual and linguistic features into the disease inference module. It inputs a multimodal feature vector, which is a fusion of facial feature vectors containing temporal information and text semantic vectors, into a classifier (such as SVM, Random Forest, or deep neural network classifier) ​​to predict the patient's mental state category and severity, providing doctors with auxiliary diagnostic references.

[0067] S303: Third Branch: 3D Facial Recognition Matching and Disease Visualization Module, such as... Figure 4 As shown This module aims to reconstruct facial videos into a three-dimensional structure and match it with a general 3D face database to achieve standardized visual representation of medical conditions. The specific technical process is as follows: (1) Three-dimensional face reconstruction and feature extraction The system uses a 3D face reconstruction algorithm to process the input facial video, extract the depth feature information of the face, and obtain detailed morphological data of the face in 3D space, laying the foundation for subsequent steps.

[0068] (2) 3D-CNN encoding, alignment and face generation related operations Encoding and Comparison: The 3D face reconstruction results are encoded using a 3D convolutional neural network (3D-CNN). 3D-CNN can effectively process 3D spatial data and extract more representative features, as shown in formula (9): (9); in, Represents the input three-dimensional facial feature body; This is a 3D convolution operation used to extract local features of facial structure and dynamic information; To correct the linear unit activation function for introducing non-linear feature mapping; For max-pooling operation to reduce feature dimension and retain the main response; For unfolding the multi-dimensional feature map into a one-dimensional vector; f is the feature vector extracted by the three-dimensional convolutional network, which is input into the three-dimensional face map matching and disease visualization module to calculate the similarity between the face and the standard face map.

[0069] Compare the encoding result with the preset general three-dimensional face map library, and use the Euclidean distance (see formula (10)) and other measurement methods to calculate the similarity between the input sample and the typical disease face map (such as "severe depression face map" and "mania attack face map"). For example, the output "cynura area depression feature and severe depression spectrum face map matching degree is 90%".

[0070] The general three-dimensional face map library is a standardized database pre-constructed by the system. The construction method is as follows: collect facial video data of clinically diagnosed patients with different types of mental illness (such as unipolar affective disorder and bipolar affective disorder) and healthy people; use three-dimensional face reconstruction algorithm to generate corresponding three-dimensional face model for each individual; then, according to the clinical diagnosis label, the model is grouped, and the spatial geometric mean of the three-dimensional model in each group is calculated to generate a "standard three-dimensional face map" representing each typical disease.

[0071] (10); Among them, The Euclidean distance between the input feature vector and the database feature vector is represented by d; is the three-dimensional feature vector obtained by the input video through the three-dimensional convolution feature extraction module; is the standard face map feature vector stored in the database; and respectively represent the values of the two vectors in the first i feature dimension.

[0072] (3) Emotion baseline correction and face map refinement (S400) Emotion baseline is the average level and feature mode of individual emotional performance in normal and stable state, which provides a reference standard for judging emotional fluctuations and identifying abnormal emotional state, and helps emotion-related research and diagnosis.

[0073] S401: Regularity analysis and baseline correction: analyze the motion trajectories of facial motion units and 68 key points over time, find the specific regularities of patients with unipolar affective disorder and bipolar affective disorder, and correct the emotion baseline of the standardized three-dimensional face accordingly to eliminate the influence of individual differences on face map analysis, so that the face map can better reflect the disease-related features.

[0074] Specifically, it includes: for all patient videos in the database, by using the time sequence of facial action unit intensity and the time sequence of 68 key point coordinates obtained when executing steps S202 and S203, a personal neutral baseline is established. For all features extracted from the neutral expression video, the statistical values are calculated, and the average intensity value of each motion unit and the average position of each key point (x, y) coordinate are calculated as the personal baseline of the patient.

[0075] For the emotional features of patients with depression and bipolar disorder, the system discovers regular differences across the population, such as finding that "AU12 activation in the unipolar depression group is significantly weaker than in the healthy group" or "brow key point motion variance is significantly higher in bipolar disorder patients". These rules are further transformed into morphological adjustment rules for three-dimensional masks: on the basis of the initial neutral average face, the disease characteristics are embedded by driving vertex displacement, such as generating a frown shape in the glabella region according to the "AU4 persistent high" rule, or constructing a more pronounced smile or open-mouth pose in the mouth area according to the "increased mouth corner motion range". The final three-dimensional mask is no longer a simple average appearance parameter, but a fusion of the most discriminative disease-related motion patterns mined from the data, thereby realizing the standardized visualization of mental illness features.

[0076] In summary, S401 essentially solidifies dynamic, subtle expression rules into static, visualized three-dimensional morphology. Using the original facial motion sequence, individual differences are eliminated by personal baseline correction to obtain pure emotional features; through specific group analysis, disease-specific motion patterns are found. Finally, it is used to guide S402 and S403 to generate a three-dimensional mask that better reflects the nature of the disease and is standardized. The disease mask generated in this way is more scientific and interpretable. The "depression mask" seen by the doctor is not just "looking sad", but quantitatively reflects "compared with healthy people, these muscle groups in the faces of patients of this type have these specific motion trends", greatly improving the value of auxiliary diagnosis. S402: Facial wrinkle refinement and mask generation: Based on the emotional baseline correction, detailed facial feature information is used for facial wrinkle refinement to generate a more realistic universal mask.

[0077] S403: In this process, the newly added 3D-CNN matches the depth information of the input video with the pre-set universal three-dimensional mask library, realizing the automatic generation of three-dimensional masks and the visual presentation of the disease.

[0078] Through the above complete technical process, this module realizes the conversion from facial video to three-dimensional mask and matching with typical disease mask, providing an intuitive, standardized visualization tool for the diagnosis and analysis of mental illnesses.

[0079] The application realizes end-to-end automatic generation from original data to multi-dimension and interpretable auxiliary diagnosis results.

[0080] The above embodiments have described the technical solutions and beneficial effects of the application in detail. It should be understood that the above description is only a specific embodiment of the application and is not used to limit the application. Any modification, supplement and equivalent replacement within the principle range of the application should be included in the protection range of the application.

Claims

1. A mental health emotion deep interaction method based on a large model, characterized in that, The method comprises the following steps: (1) collecting and preprocessing facial video data and medical text information of a mental health disease patient; (2) encoding the medical text information to obtain a medical text feature vector; extracting a facial motion unit time sequence feature vector and a 68 facial key point time sequence feature vector from the facial video data, and performing weighted fusion to obtain a facial video feature vector; (3) mapping the medical text feature vector and the facial video feature vector to the same semantic space for deep alignment to construct a cross-modal retrieval model; retrieving similar historical medical records and medical data of the patient to be tested from a known medical record database through the cross-modal retrieval model; (4) generating a dynamic fold line graph of the patient to be tested from the facial motion unit intensity sequence; generating an interpretable diagnosis report from the similar historical medical records and medical data of the patient to be tested retrieved by the cross-modal retrieval model; performing three-dimensional face reconstruction on the facial video data of the patient to be tested and comparing with a three-dimensional face library to output the similarity with a typical disease face.

2. The large model-based mental health emotional depth interaction method according to claim 1, characterized in that, In step (1), the medical text information includes diagnosis of somatic diseases, medication for somatic diseases, presence or absence of negative ideas, presence or absence of negative behaviors, whether there is a history of emotional disorder in the family, whether there is a combined mental disease, current disease and combined disease and medication, PHQ-9 scale score, Hamilton depression scale score, Hamilton anxiety scale score, and Yang's mania rating scale score.

3. The large model-based mental health emotional deep interaction method according to claim 1, characterized in that, The mental health disease patient includes a unipolar affective disorder patient and a bipolar affective disorder patient.

4. The large model-based mental health emotional depth interaction method according to claim 1, characterized in that, Step (2) comprises: (2-1) converting each text item in the medical text information into a semantic vector using a text encoder, and performing weighted fusion on the semantic vectors of all text items using a multi-head self-attention mechanism to obtain a medical text feature vector; (2-2) Extract the activation intensity value of each frame of facial motion unit from the facial video data, construct the facial motion unit activation intensity sequence and input it into the LSTM network to obtain the facial motion unit time sequence feature vector ; adopt the key point tracking algorithm based on deep learning to identify the 68 facial key point coordinate sequences in the facial video data, and encode the 68 facial key point coordinate sequences into 68 facial key point time sequence feature vectors through the 3D-CNN network ; (2-3) The face video feature vector is obtained by fusing the face image feature vector and the face video feature vector through an attention weighting mechanism. and performing fusion, obtaining a face video feature vector.

5. The large model-based mental health emotional deep interaction method according to claim 1, characterized in that, Step (3) comprises: (3-1) mapping the medical text feature vector and the facial video feature vector to the same semantic space; (3-2) training the model using a cosine similarity contrast loss function to make the facial video feature vector of the patient as close as possible to its corresponding medical text feature vector, and as far as possible from the medical text feature vectors of other patients, and obtaining a cross-modal retrieval model after training; (3-3) constructing a known medical record database from the facial video feature vectors and the medical text feature vectors of known cases; (3-4) retrieving similar historical medical records and medical data of the patient to be tested from the known medical record database through the cross-modal retrieval model.

6. The large model-based mental health emotional deep interaction method according to claim 1, characterized in that, Step (4) comprises: (4-1) extracting the activation intensity value of the facial motion unit of each frame from the facial video data, constructing a facial motion unit activation intensity sequence, inputting it into an LSTM network, and clustering and averaging the output of each time step of the LSTM network according to the disease type, taking the X-axis as the time and the Y-axis as the average activation intensity, and drawing a dynamic fold line graph for each facial motion unit of each disease type; (4-2) inputting the similar historical medical records and medical data of the patient to be tested retrieved by the cross-modal retrieval model into a fine-tuned large language model to generate an interpretable report in natural language form. (4-3) Perform three-dimensional face reconstruction based on the facial video data of the subject, and use 3D-CNN to encode the three-dimensional face reconstruction results, and calculate the similarity between the encoded results and typical disease face in the three-dimensional face database; (4-4) Visualize the results and similarity of the 3D face reconstruction.

7. The large model-based mental health emotional deep interaction method according to claim 6, characterized in that, The method for constructing a 3D face database includes: collecting facial video data of patients with different types of clinically diagnosed mental illnesses and healthy individuals; using a 3D face reconstruction algorithm to generate a corresponding 3D face model for each individual; then, grouping the models according to clinical diagnostic labels, and generating typical condition faces representing various typical conditions by calculating the spatial geometric mean of the 3D models within each group.

8. A large model-based mental health emotional depth interaction device, characterized in that, Includes the following units: The data acquisition unit collects and preprocesses facial video data and medical text information from patients with mental health disorders. The multimodal feature extraction unit encodes medical text information to obtain medical text feature vectors; it extracts the temporal feature vectors of facial motion units and 68 facial key points from facial video data, and performs weighted fusion to obtain facial video feature vectors. The cross-modal alignment and retrieval module maps medical text feature vectors and facial video feature vectors to the same semantic space for deep alignment, constructing a cross-modal retrieval model; and retrieves historical medical records and medical data similar to the subject from a known medical record database through the cross-modal retrieval model. The results generation module generates a dynamic facial polygonal map based on the temporal feature vector extracted from the facial motion unit intensity sequence of the subject; it generates an interpretable diagnostic report based on the historical medical records and medical data similar to the subject retrieved by the cross-modal retrieval model; and it performs 3D face reconstruction based on the subject's facial video data and compares it with a 3D face database to output the similarity with typical disease face patterns.

9. A large model-based mental health emotional deep interaction device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and the processor executes the computer program to implement the deep interactive method for mental health and emotions based on a large model as described in any one of claims 1-7.

Images