Multi-modal mental health prediction method, electronic equipment and medium

Abstract

The embodiment of the invention provides a multi-mode mental health prediction method, electronic equipment and a medium, and belongs to the technical field of medical science and technology and artificial intelligence. The method comprises the following steps: performing psychological health detection on target user psychological features of fused user micro-expression features, user micro-action features, user facial features, user emotion categories and user micro-mood categories to obtain psychological health prediction data of a target user. According to the embodiment of the invention, psychological health detection is carried out on the target user psychological features fused with the user micro-expression features, the user micro-action features, the user facial features, the user emotion categories and the user micro-mood categories, so that the psychological activity state of the user can be comprehensively captured; the problem that detection is inaccurate due to the fact that psychological health detection is carried out only through user micro-expressions as single basis features is solved, and the accuracy of psychological health detection is remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, is applicable to the field of medical technology, and in particular to a multimodal mental health prediction method, electronic equipment, and medium. Background Art

[0002] At present, the traditional deep learning model (Convolutional Neural Network, CNN) is usually used to identify the micro-expressions of the user in questioning, and the psychological health of the user in questioning is tested based on the micro-expressions. For example, in the medical technology scenario, a user consultation video is obtained, and the user's micro-expressions (such as lack of changes in facial expressions for a long time, raised eyebrows, drooping corners of the mouth, or suppressed smiles) are extracted from the user consultation video through CNN, and the user's micro-expressions are analyzed to identify the user's psychological health problems such as anxiety disorders. However, the feature basis of this method in psychological health testing is only the user's micro-expressions, and the feature basis is relatively single, resulting in low accuracy of psychological health testing. Therefore, how to improve the accuracy of psychological health testing has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a multimodal mental health prediction method, electronic device and medium, aiming to improve the accuracy of mental health detection.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a multimodal mental health prediction method, the method comprising:

[0005] Obtain the target user's consultation video, consultation voice and consultation text;

[0006] Obtaining a pre-trained mental health detection model; wherein the mental health detection model includes a visual feature extraction layer, an emotion recognition layer, a micro-tone recognition layer, and a mental health prediction layer;

[0007] Performing micro-expression recognition on the medical consultation video through the visual feature extraction layer to obtain user micro-expression features;

[0008] Performing micro-motion recognition on the medical consultation video through the visual feature extraction layer to obtain user micro-motion features;

[0009] Performing facial feature recognition on the consultation video through the visual feature extraction layer to obtain user facial features;

[0010] Performing emotion recognition on the medical consultation text through the emotion recognition layer to obtain the user emotion category;

[0011] Performing micro-tone recognition on the consultation voice through the micro-tone recognition layer to obtain the user micro-tone category;

[0012] The user's micro-expression features, the user's micro-movement features, the user's facial features, the user's emotion categories, and the user's micro-tone categories are feature-fused to obtain the target user's psychological features;

[0013] The mental health prediction layer performs mental health prediction on the psychological characteristics of the target user to obtain the mental health prediction data of the target user.

[0014] In some embodiments, performing mental health prediction on the psychological characteristics of the target user through the mental health prediction layer to obtain the mental health prediction data of the target user includes:

[0015] Position encoding the psychological characteristics of the target user is performed through the mental health prediction layer to obtain a psychological position feature;

[0016] Performing self-attention processing on the psychological position feature to obtain a psychological attention feature;

[0017] Identifying the mental health level of the mental attention characteristics to obtain a mental health probability;

[0018] The psychological health prediction is performed on the psychological attention characteristics according to the psychological health probability to obtain the psychological health prediction data of the target user.

[0019] In some embodiments, the step of fusing the user micro-expression features, the user micro-movement features, the user facial features, the user emotion categories, and the user micro-tone categories to obtain the target user psychological features includes:

[0020] Converting the user's micro-expression features into micro-expression text features, and obtaining micro-expression feature weights of the micro-expression text features;

[0021] Converting the user micro-motion feature into a micro-motion text feature, and obtaining a micro-motion feature weight of the micro-motion text feature;

[0022] Converting the user's facial features into facial text features, and obtaining facial feature weights of the facial text features;

[0023] Converting the user micro-tone category into a micro-tone text feature, and obtaining a micro-tone feature weight of the micro-tone text feature;

[0024] The emotion feature weights and emotion text features of the user emotion category are obtained, and the micro-expression text features, the micro-movement text features, the facial feature weights, the micro-tone feature weights and the emotion feature weights are feature-fused to obtain the psychological features of the target user.

[0025] In some embodiments, the step of performing micro-expression recognition on the consultation video through the visual feature extraction layer to obtain user micro-expression features includes:

[0026] Acquire the video frame image of the medical consultation video through the visual feature extraction layer, and segment the video frame image to obtain video frame image blocks;

[0027] Encoding the video frame image block to obtain a video frame image feature block;

[0028] Exchanging adjacent video frame feature blocks of the video frame image feature blocks to obtain a mixed video frame image feature block;

[0029] Performing micro-expression attention recognition on the mixed video frame image feature block to obtain micro-expression attention features;

[0030] The micro-expression attention feature is decoded to obtain the user micro-expression feature.

[0031] In some embodiments, the step of performing micro-motion recognition on the medical consultation video through the visual feature extraction layer to obtain user micro-motion features includes:

[0032] Obtaining a video frame feature map of the medical consultation video through the visual feature extraction layer, and performing global average pooling on the video frame feature map to obtain a video frame global feature;

[0033] Obtaining a video frame feature weight of each of the video frame global features, and determining a spatial video frame feature map according to the video frame feature weight and the video feature map;

[0034] Dividing the spatial video frame feature map by time dimension to obtain a video frame time feature map;

[0035] Performing temporal reconstruction on the video frame temporal feature graph to obtain a reconstructed video frame temporal feature graph;

[0036] Aggregating the spatial video frame feature map with the reconstructed video frame temporal feature map to obtain a target video frame feature map;

[0037] Micro-motion classification is performed on the target video frame feature map to obtain the user micro-motion features.

[0038] In some embodiments, the step of performing facial feature recognition on the consultation video through the visual feature extraction layer to obtain the user's facial features includes:

[0039] Performing facial region recognition on the medical consultation video through the visual feature extraction layer to obtain a target facial region;

[0040] Performing 3D convolution processing on the target face area to obtain facial features;

[0041] Perform facial feature recognition on the facial features to obtain the user facial features.

[0042] In some embodiments, the performing micro-tone recognition on the consultation voice by the micro-tone recognition layer to obtain the user micro-tone category includes:

[0043] Extracting the spectrum features of the consultation speech through the micro-tone recognition layer to obtain a speech spectrum graph;

[0044] Performing convolution processing on the speech spectrogram to obtain speech spectrogram features;

[0045] Performing pooling processing on the speech spectrogram features to obtain reduced-dimensional speech spectrogram features;

[0046] The reduced-dimensional speech spectrogram features are subjected to micro-tone classification to obtain the user micro-tone category.

[0047] In some embodiments, the step of performing emotion recognition on the medical consultation text through the emotion recognition layer to obtain the user emotion category includes:

[0048] Performing word embedding processing on the medical consultation text through the emotion recognition layer to obtain a medical consultation text vector;

[0049] Encoding the medical inquiry text vector to obtain medical inquiry text features;

[0050] Emotion category recognition is performed on the features of the medical consultation text to obtain the user emotion category.

[0051] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0052] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0053] The multimodal mental health prediction method, electronic device and medium proposed in the present application obtain the consultation video, consultation voice and consultation text of the target user; obtain a pre-trained mental health detection model; wherein the mental health detection model includes a visual feature extraction layer, an emotion recognition layer, a micro-tone recognition layer and a mental health prediction layer; perform micro-expression recognition on the consultation video through the visual feature extraction layer to obtain user micro-expression features; perform micro-movement recognition on the consultation video through the visual feature extraction layer to obtain user micro-movement features; perform facial feature recognition on the consultation video through the visual feature extraction layer to obtain user facial features; perform emotion recognition on the consultation text through the emotion recognition layer to obtain user emotion categories; perform micro-tone recognition on the consultation voice through the micro-tone recognition layer to obtain user micro-tone categories; perform feature fusion of user micro-expression features, user micro-movement features, user facial features, user emotion categories and user micro-tone categories to obtain psychological characteristics of the target user; perform mental health prediction on the psychological characteristics of the target user through the mental health prediction layer to obtain mental health prediction data of the target user. The present application conducts mental health detection on the target user's psychological characteristics that integrate user micro-expression features, user micro-movement features, user facial features, user emotion categories, and user micro-tone categories. It can comprehensively capture the user's psychological activity state, solve the problem of inaccurate detection caused by only using user micro-expressions as a single feature for mental health detection, and significantly improve the accuracy of mental health detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of a multimodal mental health prediction method provided in an embodiment of the present application;

[0055] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0056] Figure 3 yes Figure 1 Flow chart of step S104 in FIG.

[0057] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.

[0058] Figure 5 yes Figure 1 Flow chart of step S106 in FIG.

[0059] Figure 6 yes Figure 1 Flow chart of step S107 in FIG.

[0060] Figure 7 yes Figure 1 Flow chart of step S108 in FIG.

[0061] Figure 8 yes Figure 1 Flow chart of step S109 in FIG.

[0062] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] First, some nouns involved in this application are analyzed:

[0067] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0068] Based on this, the embodiments of the present application provide a multimodal mental health prediction method, electronic device and medium, aiming to improve the accuracy of mental health testing.

[0069] The multimodal mental health prediction method, electronic device and medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the multimodal mental health prediction method in the embodiments of the present application is described.

[0070] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0071] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, processing / interaction systems, mechatronics, etc. Artificial intelligence software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0072] The multimodal mental health prediction method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The multimodal mental health prediction method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services; the software can be an application that implements a multimodal mental health prediction method, etc., but is not limited to the above forms.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0074] Figure 1 is an optional flowchart of the multimodal mental health prediction method provided in the embodiment of the present application, Figure 1 The method may include but is not limited to steps S101 to S109.

[0075] Step S101, obtaining the target user's consultation video, consultation voice and consultation text.

[0076] Step S102, obtaining a pre-trained mental health detection model; wherein the mental health detection model includes a visual feature extraction layer, a micro-tone recognition layer, an emotion recognition layer and a mental health prediction layer.

[0077] Step S103, performing micro-expression recognition on the medical consultation video through the visual feature extraction layer to obtain user micro-expression features.

[0078] Step S104, performing micro-motion recognition on the medical consultation video through the visual feature extraction layer to obtain user micro-motion features.

[0079] Step S105, performing facial feature recognition on the consultation video through the visual feature extraction layer to obtain the user's facial features.

[0080] Step S106, performing emotion recognition on the medical consultation text through the emotion recognition layer to obtain the user emotion category.

[0081] Step S107, performing micro-tone recognition on the consultation voice through the micro-tone recognition layer to obtain the user's micro-tone category.

[0082] Step S108, performing feature fusion on user micro-expression features, user micro-movement features, user facial features, user emotion categories and user micro-tone categories to obtain psychological features of the target user.

[0083] Step S109, performing mental health prediction on the psychological characteristics of the target user through the mental health prediction layer to obtain mental health prediction data of the target user.

[0084] Steps S101 to S109 shown in the embodiment of the present application are performed by obtaining a consultation video, consultation voice and consultation text of the target user; obtaining a pre-trained mental health detection model; wherein the mental health detection model includes a visual feature extraction layer, an emotion recognition layer, a micro-tone recognition layer and a mental health prediction layer; performing micro-expression recognition on the consultation video through the visual feature extraction layer to obtain user micro-expression features; performing micro-movement recognition on the consultation video through the visual feature extraction layer to obtain user micro-movement features; performing facial feature recognition on the consultation video through the visual feature extraction layer to obtain user facial features; performing emotion recognition on the consultation text through the emotion recognition layer to obtain user emotion categories; performing micro-tone recognition on the consultation voice through the micro-tone recognition layer to obtain user micro-tone categories; performing feature fusion of user micro-expression features, user micro-movement features, user facial features, user emotion categories and user micro-tone categories to obtain psychological features of the target user; performing mental health prediction on the psychological features of the target user through the mental health prediction layer to obtain mental health prediction data of the target user. The present application conducts mental health detection on the target user's psychological characteristics that integrate user micro-expression features, user micro-movement features, user facial features, user emotion categories, and user micro-tone categories. It can comprehensively capture the user's psychological activity state, solve the problem of inaccurate detection caused by only using user micro-expressions as a single feature for mental health detection, and significantly improve the accuracy of mental health detection.

[0085] In step S101 of some embodiments, specifically, the consultation video of the target user refers to a consultation dialogue video between the user and the psychologist when the user undergoes a mental health assessment.

[0086] Specifically, the consultation voice refers to the conversation voice between the user and the psychologist extracted from the consultation video.

[0087] Specifically, the consultation text refers to the text of the conversation between the user and the psychologist extracted from the consultation video and the text of the user's mental health self-assessment and peer assessment scale (such as the concise MINI scale, DSM-V standard clinical examination manual, etc.).

[0088] In step S102 of some embodiments, specifically, the pre-trained mental health detection model includes a visual feature extraction layer, an emotion recognition layer, a micro-tone recognition layer and a mental health prediction layer.

[0089] Specifically, the visual feature extraction layer includes a micro-expression recognition network layer, a micro-motion recognition network layer and a facial feature recognition network layer, and the micro-expression recognition network layer is a network layer composed of an Encoder and a Decoder; the micro-motion recognition layer is composed of a Resnet (Residual Neural Network) architecture; the facial feature recognition network layer is composed of ViViT (Video Vision Transformer), which is used to recognize micro-expressions, micro-motions and facial features of the consultation video; the emotion recognition layer is a Transformer network, which is used to recognize emotions in the consultation text; the micro-tone recognition layer is a convolutional network, which is used to recognize micro-tones in the consultation voice.

[0090] See also Figure 2 In some embodiments, step S103 includes but is not limited to steps S201 to S205:

[0091] Step S201, obtaining a video frame image of a medical consultation video through a visual feature extraction layer, and segmenting the video frame image to obtain video frame image blocks.

[0092] Step S202, encoding the video frame image block to obtain the video frame image feature block.

[0093] Step S203, swapping adjacent video frame feature blocks of the video frame image feature blocks to obtain a mixed video frame image feature block.

[0094] Step S204, performing micro-expression attention recognition on the mixed video frame image feature block to obtain micro-expression attention features.

[0095] Step S205, decoding the micro-expression attention features to obtain user micro-expression features.

[0096] In step S201 of some embodiments, specifically, the video frame image is used to represent static screen images of the video at different moments.

[0097] Specifically, the consultation video may be decomposed into multiple image frames by performing pre-emphasis, frame division, windowing and other processing, and these image frames are usually saved as independent image information in an image format (such as JPEG, BMP) and the like.

[0098] Specifically, the video frame image block collectively represents each frame image through a plurality of non-overlapping image blocks.

[0099] Specifically, based on the three dimensions of the video frame image, namely, image height, image width and image channel, the video frame image is divided into a plurality of non-overlapping image blocks, and each image block is represented by the three dimensions of image height, image width and image channel. Among them, the image channel dimension contains the color of the image or the characteristic information of the image.

[0100] For example, a video frame image may be segmented into 16 video frame image blocks, each video frame image block representing an area of ​​the user's face, such as the user's eyes, nose, or mouth.

[0101] In step S202 of some embodiments, specifically, the video frame image feature block refers to an image region extracted from the video frame image block that is related to the user's micro-expression features, such as the texture of the user's facial features and the shape changes of the user's facial features.

[0102] Specifically, the encoder includes a multi-head attention mechanism layer and a multi-layer perceptron, and includes a normalization layer before the multi-head attention mechanism layer and the multi-layer perceptron layer.

[0103] Specifically, the encoder is designed to map each video frame image block to a low-dimensional latent vector space to generate a latent vector that not only contains the visual content of the video frame image block, but also incorporates the spatial position information of the video frame image block in the image through position encoding.

[0104] Furthermore, the encoding process begins by concatenating the hidden vectors of the video frame image blocks into a matrix, which contains the hidden vector representations of all video frame image blocks. Subsequently, the matrix is ​​input into the encoder, which consists of a normalization layer, a multi-head attention mechanism, and a multi-layer perceptron. The matrix containing the hidden vector can be normalized by the normalization layer; the multi-head attention machine takes into account the information of other blocks when processing each video frame image block, thereby capturing the dependencies between the video frame image blocks; the multi-layer perceptron further extracts and transforms the features of the video frame image blocks, thereby realizing the conversion of the hidden vector into a video frame image feature block.

[0105] In this embodiment, by encoding the video frame image block, the obtained video frame image feature block not only contains the visual features of the video frame image block, but also integrates the spatial relationship and contextual information of the video frame image block through the processing of the encoder, thereby providing a rich feature basis for subsequent micro-expression recognition.

[0106] In step S203 of some embodiments, specifically, the mixed video frame image feature block refers to an image feature block containing different facial expression features.

[0107] Specifically, image feature blocks may be exchanged between consecutive video frame image feature blocks to simulate changes in micro-expressions.

[0108] For example, if a video frame image feature block is a user smiling, and a next video frame image feature block is a user frowning, by exchanging the frowning and smiling image feature blocks, a mixed video frame image feature block can be obtained, which contains mixed features of the user smiling and the user frowning.

[0109] In this embodiment, by exchanging adjacent video frame feature blocks, a mixed video frame image feature block is obtained, which can enhance the visual feature extraction layer's ability to recognize changes in user micro-expressions, thereby facilitating subsequent improvement in the accuracy of micro-expression recognition.

[0110] In step S204 of some embodiments, specifically, the micro-expression attention feature refers to a key feature related to the micro-expression.

[0111] Specifically, the optical flow vector between the feature blocks of the two mixed video frames is first calculated to obtain the user's facial motion vector. The similarity between the feature blocks of the two mixed video frames is compared through the block of interest. The facial motion vector is processed through the attention mechanism according to the similarity to obtain the micro-expression attention feature, which can measure the weight of each mixed video frame image feature block. The higher the weight, the higher the correlation between the mixed video frame image feature block and the user's facial expression.

[0112] For example, if rapid changes in the user's eye area are detected, the attention mechanism may focus on the user's eye area.

[0113] In this embodiment, micro-expression attention features are obtained by performing micro-expression attention recognition on the mixed video frame image feature blocks, which can capture slight changes in the user's facial expressions between consecutive video frames, thereby facilitating the subsequent improvement of the accuracy of micro-expression recognition.

[0114] In step S205 of some embodiments, specifically, the user micro-expression feature refers to a small facial movement feature of the user in a short period of time (such as 0.25-0.5 seconds).

[0115] For example, user micro-expression features may be small expressions related to mental health assessment, such as lack of changes in facial expressions for a long time, raising of the inner side of eyebrows, and suppression of smiles.

[0116] Specifically, the decoder can convert the micro-expression attention features into the visualization features of the user's micro-expressions. First, the micro-expression attention features are decoded into initial decoded features with the same dimension as the video frame image block through the decoding function; secondly, the initial decoded features are converted into intermediate signal features through the linear layer; finally, the intermediate signal features are reshaped to generate a user micro-expression image with the same height and width as the video frame image, and then the micro-expression features of the user micro-expression image are extracted to obtain the user micro-expression features.

[0117] In this embodiment, by decoding the micro-expression attention features, the user's micro-expression features are obtained, and the compressed potential micro-expression features can be restored to an image that is consistent with the video frame image space, color, height, and style, and the image contains detailed features of the user's micro-expressions. The user's micro-expression features are obtained from the image, which effectively improves the accuracy of micro-expression recognition.

[0118] See also Figure 3 In some embodiments, step S104 includes but is not limited to steps S301 to S306:

[0119] Step S301, obtaining a video frame feature map of the medical consultation video through a visual feature extraction layer, and performing global average pooling on the video frame feature map to obtain a video frame global feature.

[0120] Step S302, obtaining the video frame feature weight of the global feature of each video frame, and determining the spatial video frame feature map according to the video frame feature weight and the video feature map.

[0121] Step S303, dividing the spatial video frame feature map by time dimension to obtain a video frame time feature map.

[0122] Step S304, performing temporal reconstruction on the video frame temporal feature graph to obtain a reconstructed video frame temporal feature graph.

[0123] Step S305, aggregating the spatial video frame feature map and the reconstructed video frame temporal feature map to obtain a target video frame feature map.

[0124] Step S306, performing micro-motion classification on the target video frame feature map to obtain user micro-motion features.

[0125] In step S301 of some embodiments, specifically, the Resnet architecture includes a Squeeze-and-Excitation network (SE) and a Temporal Shift Module (TSM). SE is used to enhance the attention of the Resnet architecture to important features while suppressing unimportant features; TSM is used to enhance the Resnet architecture's understanding of the temporal relationship in time series data.

[0126] Specifically, the consultation video is first mapped into a feature map of each frame through the Resnet architecture.

[0127] Specifically, the average value of each channel of each video frame feature map is calculated through the SE module to perform global average pooling processing, so as to compress the video frame feature map of each channel into a single value to determine the global features of the video frame.

[0128] In this embodiment, by performing global average pooling on the video frame feature map, the feature dimension of the video frame feature map can be increased while retaining important global information, thereby obtaining the global features of the video frame feature map.

[0129] In step S302 of some embodiments, specifically, the video frame feature weight is used to represent more important image features in the video frame, so as to highlight the image area related to micro-motion in the consultation video frame image.

[0130] Specifically, the video frame global features are compressed to a lower dimension through the first fully connected layer of SE, aiming to capture the correlation of global features of different video frames in a lower dimension, and the compressed global features are excitation processed through the second fully connected layer to restore the compressed global feature dimension to the original dimension to generate the video frame feature weights of each video frame global feature.

[0131] Furthermore, the video frame feature weights and the video frame global features are element-by-element multiplied to obtain a weighted spatial video frame feature map.

[0132] In this embodiment, by obtaining the video frame feature weights of the global features of each video frame and determining the spatial video frame feature map based on the video frame feature weights and the video feature map, it is possible to focus on the important features of the global features of the video frame and suppress unimportant features, which helps to improve the accuracy of subsequent micro-motion recognition.

[0133] In step S303 of some embodiments, specifically, the spatial video frame feature map is segmented into multiple blocks along the time dimension to obtain a video frame time feature map, wherein each block represents a spatial video frame feature map of a time step.

[0134] In this embodiment, the spatial video frame feature map is divided into time dimensions to segment the feature map along the time axis, so as to facilitate the subsequent analysis of changes in user micro-movements in the video over time.

[0135] In step S304 of some embodiments, specifically, each video frame temporal feature map is offset in the time dimension to simulate changes in the time series. For example, if a block contains consecutive video frame temporal feature maps, TSM will move these frames forward or backward by one time step.

[0136] Furthermore, the offset video frame temporal feature map is input into three convolutional layers with different convolution kernel sizes (such as 1x1, 3x3, and 1x1), and the outputs of these convolutional layers are residually connected with the spatial video frame feature map to obtain the reconstructed video frame temporal feature map.

[0137] In this embodiment, by temporally reconstructing the temporal feature map of the video frame, TSM can effectively capture the temporal dynamic information between video frame images without adding additional parameters, and provides feature data support for capturing user micro-motions, thereby facilitating the subsequent improvement of the accuracy of micro-motion recognition.

[0138] In step S305 of some embodiments, specifically, by aggregating the spatial video frame feature map and the reconstructed video frame temporal feature map through global average pooling, a target video frame feature map that integrates spatial and temporal information can be obtained. This feature map contains key user micro-motion information in the video and can also reflect the changes of this micro-motion information over time.

[0139] In this embodiment, the spatial video frame feature map and the reconstructed video frame temporal feature map are aggregated to obtain the target video frame feature map, which not only contains spatial information, but also integrates the micro-motion features in the temporal dimension, providing richer feature information related to user micro-motions.

[0140] In step S306 of some embodiments, specifically, the user micro-motion feature may be a slight change in the user's facial features.

[0141] For example, in the medical field, user micro-motion features may be crying, lowering the head, turning the head, and wandering eyes, which are actions related to mental health assessment.

[0142] Specifically, the ReLU activation function, weight parameters and bias parameters of the fully connected layer are obtained, and the micro-motion classification is performed on the target video frame feature map according to the ReLU activation function, weight parameters and bias parameters to obtain the final user micro-motion features.

[0143] In this embodiment, by performing micro-motion classification on the target video frame feature map, the user's subtle movements in the video can be identified and classified, which is convenient for subsequently improving the accuracy of mental health prediction.

[0144] See also Figure 4 In some embodiments, step S105 includes but is not limited to steps S401 to S403:

[0145] Step S401, performing face region recognition on the medical consultation video through the visual feature extraction layer to obtain the target face region.

[0146] Step S402, performing 3D convolution processing on the target face area to obtain facial features.

[0147] Step S403, performing facial feature recognition on the facial features to obtain the user's facial features.

[0148] In step S401 of some embodiments, specifically, the target face area refers to an area related to the user's face in the medical consultation video.

[0149] Specifically, it is achieved through a face detection algorithm (such as Haar feature), and after the face area is identified, the target face area is obtained.

[0150] In step S402 of some embodiments, specifically, the user's facial features include static features such as facial geometry features and skin texture features.

[0151] Specifically, the target face area is converted into image tokens, the time, height and width information of the image tokens are obtained through the tubular unit of VIVIT, and the tubular unit is encoded using 3D convolution to obtain the temporal features of the user's face in the target face area. The temporal and spatial information can be integrated in the token generation stage, effectively capturing the temporal and spatial features in the target face area and avoiding the phenomenon of information image splitting.

[0152] In this embodiment, by performing 3D convolution processing on the target face area, facial features containing time and space information can be extracted from the face area, which is convenient for subsequently improving the accuracy of facial feature recognition.

[0153] In step S403 of some embodiments, specifically, the user's facial features refer to descriptions of the user's skin complexion and skin static features.

[0154] For example, the user's facial features may be pale complexion, sallow complexion, frown lines between eyebrows, skin folliculitis, skin neurodermatitis, and other features related to mental health detection.

[0155] Specifically, the facial features are input into the fully connected layer of VIVIT, and the facial features are classified by the Sigmoid function of the fully connected layer. Multiple probabilities between 0 and 1 are output corresponding to the probability of identifying the corresponding user facial features, and the labels exceeding the facial feature threshold (such as 0.5) are regarded as the identified user facial features.

[0156] In this embodiment, by performing facial feature recognition on human face features, the skin condition and texture of the face in the video can be captured, effectively improving the accuracy of subsequent mental health testing.

[0157] See also Figure 5 In some embodiments, step S106 includes but is not limited to steps S501 to S503:

[0158] Step S501, word embedding processing is performed on the medical consultation text through the emotion recognition layer to obtain a medical consultation text vector.

[0159] Step S502: Encode the medical inquiry text vector to obtain medical inquiry text features.

[0160] Step S503: perform emotion category recognition on the features of the medical inquiry text to obtain the user emotion category.

[0161] In step S501 of some embodiments, specifically, the medical inquiry text vector is text represented by points in a continuous vector space.

[0162] Specifically, word embedding is a method of converting words in a medical consultation text into points in a continuous vector space. The word embedding processing of the medical consultation text can be achieved by performing an Embedding operation on the medical consultation text.

[0163] In this embodiment, the word embedding process is performed on the medical consultation text through the emotion recognition layer, which can capture the semantic relationship between words, and the medical consultation text vector not only contains the information of the text words themselves, but also may implicitly indicate the importance of the words in the expression of emotions.

[0164] In step S502 of some embodiments, specifically, the medical inquiry text feature refers to medical inquiry text contextual semantic information related to user emotions.

[0165] Specifically, the contextual information in the medical inquiry text vector is captured through the self-attention layer of the Transformer, and the high-dimensional contextual information is mapped to low-dimensional information through the linear layer, thereby capturing the contextual semantic features of the medical inquiry text.

[0166] In step S503 of some embodiments, specifically, the user emotion categories include but are not limited to happiness, sadness, anger, surprise, and fatigue.

[0167] Specifically, according to the output weight, output bias and output hidden state parameters of the Transformer classification layer, the probability of the emotion category corresponding to the text feature is diagnosed through the Softmax function. When the emotion category probability is greater than the emotion category threshold (such as 0.6), the emotion corresponding to the emotion category probability is used as the user emotion category.

[0168] In this embodiment, by performing emotion category recognition on the features of the medical consultation text, the most likely emotion category can be judged according to the weight of the features of the medical consultation text, thereby determining the user's emotion category and providing emotion data support for subsequent mental health testing.

[0169] See also Figure 6 In some embodiments, step S107 includes but is not limited to steps S601 to S604:

[0170] Step S601, extracting spectral features of the consultation speech through the micro-tone recognition layer to obtain a speech spectrogram.

[0171] Step S602, performing convolution processing on the speech spectrogram to obtain speech spectrogram features.

[0172] Step S603, performing pooling processing on the speech spectrogram features to obtain reduced-dimensional speech spectrogram features.

[0173] Step S604, performing micro-tone classification on the dimension-reduced speech spectrogram features to obtain a user micro-tone category.

[0174] In step S601 of some embodiments, specifically, the speech spectrogram can display the intensity distribution of speech information at different frequencies, so as to visually reflect the characteristics of the consultation speech, such as timbre, pitch and intensity.

[0175] Specifically, the voice signal of the consultation speech can be converted into a speech spectrogram through the Mel-frequency cepstral coefficient.

[0176] In step S602 of some embodiments, specifically, a convolution operation may be performed on the speech spectrogram according to a convolution kernel weight and a convolution kernel bias to identify and extract speech spectrogram features of the speech spectrogram.

[0177] In step S603 of some embodiments, specifically, the maximum pooling process may be performed on the speech spectrogram features to reduce the dimension of the data and thereby reduce the computational complexity, thereby obtaining the reduced-dimensional speech spectrogram features.

[0178] In this embodiment, by performing pooling processing on the speech spectrogram features, the dimension of the speech spectrogram can be reduced to remove unimportant features while retaining the most important feature information.

[0179] In step S604 of some embodiments, specifically, the micro-tone categories include but are not limited to hesitation, excessive pauses, vibrato, and dullness, etc., which are characteristic information related to mental health detection.

[0180] For example, if the model identifies that a user hesitates when talking about a psychology-related topic, it may indicate that the user has psychological stress or anxiety about this topic.

[0181] Specifically, micro-tone classification is a fine-grained speech emotion recognition task that is used to identify and distinguish subtle emotional changes in the user's voice. It usually involves the fully connected layer of a deep neural network to identify different micro-tone categories. By analyzing the features of the reduced-dimensional speech spectrogram, the model can predict the user's micro-tone category during the consultation process.

[0182] In this embodiment, by performing micro-tone classification on the dimensionality-reduced speech spectrogram features, micro-tone feature support can be provided for subsequent mental health testing, which can effectively improve the accuracy of mental health testing.

[0183] See also Figure 7 In some embodiments, step S108 includes but is not limited to steps S701 to S705:

[0184] Step S701, converting the user's micro-expression features into micro-expression text features, and obtaining the micro-expression feature weights of the micro-expression text features.

[0185] Step S702: convert the user micro-motion features into micro-motion text features, and obtain the micro-motion feature weights of the micro-motion text features.

[0186] Step S703: convert the user's facial features into facial text features, and obtain the facial feature weights of the facial text features.

[0187] Step S704: convert the user micro-tone category into a micro-tone text feature, and obtain the micro-tone feature weight of the micro-tone text feature.

[0188] Step S705, obtain the emotion feature weights and emotion text features of the user emotion category, and fuse the micro-expression text features, micro-movement text features, facial text features, emotion text features and micro-tone text features according to the micro-expression feature weights, micro-movement feature weights, facial feature weights, micro-tone feature weights and emotion feature weights to obtain the target user's psychological characteristics.

[0189] In step S701 of some embodiments, specifically, before performing the mental health test, user features from different sources need to be converted into text features for unified processing and analysis.

[0190] Specifically, the user's micro-expression features can be mapped to a vector in a high-dimensional space through word embedding (such as Word Embedding). This vector can capture the semantic information of the feature to obtain the micro-expression text features.

[0191] Specifically, the micro-expression feature weight is used to measure the importance of micro-expression text features in mental health detection, and the micro-expression feature weight can be obtained through the attention mechanism.

[0192] In step S702 of some embodiments, specifically, the user micro-motion feature can be mapped to a vector in a high-dimensional space through word embedding (such as Word Embedding), and this vector can capture the semantic information of the feature to obtain the micro-motion text feature.

[0193] Specifically, the micro-movement feature weight is used to measure the importance of micro-movement text features in mental health detection, and the micro-movement feature weight can be obtained through the attention mechanism.

[0194] In step S703 of some embodiments, specifically, the user's facial features can be mapped to a vector in a high-dimensional space through word embedding (such as Word Embedding), and this vector can capture the semantic information of the features to obtain facial text features.

[0195] Specifically, facial feature weights are used to measure the importance of facial text features in mental health detection, and facial feature weights can be obtained through the attention mechanism.

[0196] In step S704 of some embodiments, specifically, the user micro-tone category can be mapped to a vector in a high-dimensional space through word embedding (such as Word Embedding), and this vector can capture the semantic information of the feature to obtain the micro-tone text feature.

[0197] Specifically, the micro-tone feature weight is used to measure the importance of micro-tone text features in mental health detection, and the micro-tone feature weight can be obtained through the attention mechanism.

[0198] In step S705 of some embodiments, specifically, the user emotion category can be mapped to a vector in a high-dimensional space through word embedding (such as Word Embedding), and this vector can capture the semantic information of the feature to obtain the emotional text feature.

[0199] Specifically, the emotion feature weight is used to measure the importance of emotion text features in mental health detection, and the emotion feature weight can be obtained through the attention mechanism.

[0200] Specifically, the psychological characteristics of the target user are characteristics that integrate the user's micro-expression text features, micro-action text features, facial text features, emotion text features, and micro-tone text features.

[0201] For example, in a mental health assessment scenario, the target user's psychological characteristics can identify the user's micro-expressions as lack of change in facial expressions for a long time, raised inner eyebrows, and suppressed smiles; micro-movements as crying, lowering the head, turning the head, and wandering eyes; facial recognition as pale complexion, sallow complexion, frown lines between the eyebrows, skin folliculitis, and skin neurodermatitis; emotions as low mood, easy fatigue, and loss of interest; micro-tones as hesitation, too many pauses, tremors, and dullness, etc.

[0202] Specifically, the psychological characteristics of the target users can be obtained by weighted summation of different user characteristics.

[0203] In this embodiment, the psychological characteristics of the target user are obtained by fusing the micro-expression text features, micro-movement text features, facial text features, emotion text features and micro-tone text features. The multi-dimensional representation of the user's micro-expressions, micro-movements, facial features, micro-tones and emotional states is integrated to more comprehensively display the user's psychological state. This solves the problem that the traditional method of psychological health testing is based only on the user's micro-expressions and the feature basis is relatively single. This helps to make more accurate judgments and provide more personalized care during the psychological health testing process.

[0204] See also Figure 8 In some embodiments, step S109 includes but is not limited to steps S801 to 804:

[0205] Step S801, position encoding the psychological characteristics of the target user through the psychological health prediction layer to obtain psychological position characteristics.

[0206] Step S802, performing self-attention processing on the psychological position feature to obtain a psychological attention feature.

[0207] Step S803, identifying the mental health level of the psychological attention characteristics to obtain the mental health probability.

[0208] Step S804, performing mental health prediction on the psychological attention characteristics according to the mental health probability to obtain mental health prediction data of the target user.

[0209] In step S801 of some embodiments, specifically, the psychological characteristics of the target user can be positionally encoded through the psychological health prediction layer to encode the position information of the psychological characteristics of the target user into a feature vector to provide the position information of the psychological characteristic words in the sequence.

[0210] In step S802 of some embodiments, specifically, a query (Query), key (Key), and value (Value) vector of the psychological location feature is obtained, and an attention score is calculated for the vector to measure the importance of each feature in the psychological location feature. Finally, the attention score is activated through a linear layer to output the psychological attention feature.

[0211] In this embodiment, by performing self-attention processing on the psychological position features, it is possible to focus on the dependencies between different psychological features, which helps to subsequently improve the accuracy of mental health testing.

[0212] In step S803 of some embodiments, specifically, the mental health prediction layer can predict the user's mental health status based on the psychological attention characteristics. The fully connected layer in the mental health prediction layer will output a probability value for the psychological attention characteristics, indicating the possibility that the user is in a certain mental illness state. This probability value is called the mental health probability.

[0213] In step S804 of some embodiments, specifically, the mental health prediction data is used to characterize data of whether the user's psychology is in a healthy state or an unhealthy state, including but not limited to the user's mental health, the user's psychological anxiety, and the user's psychological depression.

[0214] Specifically, the mental state category whose mental health probability is less than the mental health threshold is used as the target mental state category for mental health detection.

[0215] For example, when the category probability of identifying that the user is in a psychological anxiety state is 0.6, which is greater than the psychological health threshold of 0.5, it means that the user is in a psychological anxiety state, and a psychologist can provide psychological counseling for the user to ensure the user's mental health.

[0216] In this embodiment, mental health prediction is performed on psychological attention characteristics based on the mental health probability to obtain mental health prediction data of the target user, which can provide a quantitative indicator for medical professionals to help them evaluate the user's mental health status and provide users with personalized mental health assessments and intervention suggestions, which is of great significance for promoting mental health and preventing psychological problems.

[0217] The embodiment of the present application obtains the consultation video, consultation voice and consultation text of the target user; obtains a pre-trained mental health detection model; wherein the mental health detection model includes a visual feature extraction layer, an emotion recognition layer, a micro-tone recognition layer and a mental health prediction layer; performs micro-expression recognition on the consultation video through the visual feature extraction layer to obtain user micro-expression features; performs micro-movement recognition on the consultation video through the visual feature extraction layer to obtain user micro-movement features; performs facial feature recognition on the consultation video through the visual feature extraction layer to obtain user facial features; performs emotion recognition on the consultation text through the emotion recognition layer to obtain user emotion categories; performs micro-tone recognition on the consultation voice through the micro-tone recognition layer to obtain user micro-tone categories; performs feature fusion of user micro-expression features, user micro-movement features, user facial features, user emotion categories and user micro-tone categories to obtain psychological features of the target user; performs mental health prediction on the psychological features of the target user through the mental health prediction layer to obtain mental health prediction data of the target user. The present application conducts mental health detection on the target user's psychological characteristics that integrate user micro-expression features, user micro-movement features, user facial features, user emotion categories, and user micro-tone categories. It can comprehensively capture the user's psychological activity state, solve the problem of inaccurate detection caused by only using user micro-expressions as a single feature for mental health detection, and significantly improve the accuracy of mental health detection.

[0218] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above multi-modal mental health prediction method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0219] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0220] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0221] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 902 can store processing systems and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the multimodal mental health prediction method of the embodiment of the present application;

[0222] Input / output interface 903, used to implement information input and output;

[0223] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0224] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0225] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0226] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned multimodal mental health prediction method.

[0227] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0228] The multimodal mental health prediction method, multimodal mental health prediction device, electronic device and storage medium provided in the embodiments of the present application obtain the consultation video, consultation voice and consultation text of the target user; obtain a pre-trained mental health detection model; wherein the mental health detection model includes a visual feature extraction layer, an emotion recognition layer, a micro-tone recognition layer and a mental health prediction layer; perform micro-expression recognition on the consultation video through the visual feature extraction layer to obtain user micro-expression features; perform micro-movement recognition on the consultation video through the visual feature extraction layer to obtain user micro-movement features; perform facial feature recognition on the consultation video through the visual feature extraction layer to obtain user facial features; perform emotion recognition on the consultation text through the emotion recognition layer to obtain user emotion categories; perform micro-tone recognition on the consultation voice through the micro-tone recognition layer to obtain user micro-tone categories; perform feature fusion of user micro-expression features, user micro-movement features, user facial features, user emotion categories and user micro-tone categories to obtain psychological features of the target user; perform mental health prediction on the psychological features of the target user through the mental health prediction layer to obtain mental health prediction data of the target user. The present application conducts mental health detection on the target user's psychological characteristics that integrate user micro-expression features, user micro-movement features, user facial features, user emotion categories, and user micro-tone categories. It can comprehensively capture the user's psychological activity state, solve the problem of inaccurate detection caused by only using user micro-expressions as a single feature for mental health detection, and significantly improve the accuracy of mental health detection.

[0229] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0230] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

Claims

1. A multimodal mental health prediction method, characterized in that: The method comprises: Obtain the target user's consultation video, consultation voice and consultation text; Obtaining a pre-trained mental health detection model; wherein the mental health detection model includes a visual feature extraction layer, an emotion recognition layer, a micro-tone recognition layer, and a mental health prediction layer; Performing micro-expression recognition on the medical consultation video through the visual feature extraction layer to obtain user micro-expression features; Performing micro-motion recognition on the medical consultation video through the visual feature extraction layer to obtain user micro-motion features; Performing facial feature recognition on the consultation video through the visual feature extraction layer to obtain user facial features; Performing emotion recognition on the medical consultation text through the emotion recognition layer to obtain the user emotion category; Performing micro-tone recognition on the consultation voice through the micro-tone recognition layer to obtain the user micro-tone category; The user's micro-expression features, the user's micro-movement features, the user's facial features, the user's emotion categories, and the user's micro-tone categories are feature-fused to obtain the target user's psychological features; The mental health prediction layer performs mental health prediction on the psychological characteristics of the target user to obtain the mental health prediction data of the target user.

2. The method according to claim 1, characterized in that: The mental health prediction layer performs mental health prediction on the psychological characteristics of the target user to obtain the mental health prediction data of the target user, including: Position encoding the psychological characteristics of the target user is performed through the mental health prediction layer to obtain a psychological position feature; Performing self-attention processing on the psychological position feature to obtain a psychological attention feature; Identifying the mental health level of the mental attention characteristics to obtain a mental health probability; The psychological health prediction is performed on the psychological attention characteristics according to the psychological health probability to obtain the psychological health prediction data of the target user.

3. The method according to claim 1, characterized in that The step of fusing the user's micro-expression features, the user's micro-movement features, the user's facial features, the user's emotion categories, and the user's micro-tone categories to obtain the target user's psychological features includes: Converting the user's micro-expression features into micro-expression text features, and obtaining micro-expression feature weights of the micro-expression text features; Converting the user micro-motion feature into a micro-motion text feature, and obtaining a micro-motion feature weight of the micro-motion text feature; Converting the user's facial features into facial text features, and obtaining facial feature weights of the facial text features; Converting the user micro-tone category into a micro-tone text feature, and obtaining a micro-tone feature weight of the micro-tone text feature; The emotion feature weights and emotion text features of the user emotion category are obtained, and the micro-expression text features, the micro-movement text features, the facial feature weights, the micro-tone feature weights and the emotion feature weights are feature-fused to obtain the psychological features of the target user.

4. The method according to claim 1, characterized in that: The step of performing micro-expression recognition on the consultation video through the visual feature extraction layer to obtain user micro-expression features includes: Acquire the video frame image of the medical consultation video through the visual feature extraction layer, and segment the video frame image to obtain video frame image blocks; Encoding the video frame image block to obtain a video frame image feature block; Exchanging adjacent video frame feature blocks of the video frame image feature blocks to obtain a mixed video frame image feature block; Performing micro-expression attention recognition on the mixed video frame image feature block to obtain micro-expression attention features; The micro-expression attention feature is decoded to obtain the user micro-expression feature.

5. The method according to claim 1, characterized in that The step of performing micro-motion recognition on the medical consultation video through the visual feature extraction layer to obtain user micro-motion features includes: Obtaining a video frame feature map of the medical consultation video through the visual feature extraction layer, and performing global average pooling on the video frame feature map to obtain a video frame global feature; Obtaining a video frame feature weight of each of the video frame global features, and determining a spatial video frame feature map according to the video frame feature weight and the video feature map; Dividing the spatial video frame feature map by time dimension to obtain a video frame time feature map; Performing temporal reconstruction on the video frame temporal feature graph to obtain a reconstructed video frame temporal feature graph; Aggregating the spatial video frame feature map with the reconstructed video frame temporal feature map to obtain a target video frame feature map; Micro-motion classification is performed on the target video frame feature map to obtain the user micro-motion features.

6. The method according to claim 1, characterized in that The step of performing facial feature recognition on the consultation video through the visual feature extraction layer to obtain the user's facial features includes: Performing facial region recognition on the medical consultation video through the visual feature extraction layer to obtain a target facial region; Performing 3D convolution processing on the target face area to obtain facial features; Perform facial feature recognition on the facial features to obtain the user facial features.

7. The method according to any one of claims 1 to 6, characterized in that: The step of performing micro-tone recognition on the consultation voice through the micro-tone recognition layer to obtain the user micro-tone category includes: Extracting the spectrum features of the consultation speech through the micro-tone recognition layer to obtain a speech spectrum graph; Performing convolution processing on the speech spectrogram to obtain speech spectrogram features; Performing pooling processing on the speech spectrogram features to obtain reduced-dimensional speech spectrogram features; The reduced-dimensional speech spectrogram features are subjected to micro-tone classification to obtain the user micro-tone category.

8. The method according to any one of claims 1 to 6, characterized in that: The emotion recognition layer is used to perform emotion recognition on the medical consultation text to obtain the user emotion category, including: Performing word embedding processing on the medical consultation text through the emotion recognition layer to obtain a medical consultation text vector; Encoding the medical inquiry text vector to obtain medical inquiry text features; Emotion category recognition is performed on the features of the medical consultation text to obtain the user emotion category.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the multimodal mental health prediction method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodal mental health prediction method according to any one of claims 1 to 8 is implemented.

Images