Multi-modal emotion recognition method and psychological intervention system

Abstract

The invention discloses a multi-modal emotion recognition method and a psychological intervention system in the technical field of image data processing, and aims to solve the problem that single-modal recognition cannot comprehensively reflect the real emotion state of an individual in the prior art. The method comprises the steps of obtaining continuous frame image information according to video clip processing; detecting a face feature point sequence corresponding to the continuous frame image information; performing data enhancement to obtain a first feature vector; performing weighting processing to obtain a second feature vector; performing joint processing through an expression and feature point joint module to obtain an image feature vector; extracting an audio feature vector; and performing fusion processing through an audio-visual tensor product fusion module to obtain a final fusion feature vector, and outputting an emotion recognition result of the target person by using a multi-mode fusion emotion recognition network model based on a Transform architecture. The method is not limited to a single static moment, but can track and capture the emotion dynamic change in real time, so that the response capability to the emotion dynamic state is enhanced.

Description

Technical Field

[0001] The invention relates to a multimodal emotion recognition method and a psychological intervention system, and belongs to the technical field of image data processing. Background Art

[0002] With the rapid development of society, students are facing more and more psychological pressure, such as study pressure, social anxiety, etc., which leads to negative emotions. Long-term emotional distress can lead to behavioral problems, learning disabilities, and even more serious mental health problems. At the same time, long-term neglect of emotional health has been identified as a key factor leading to various mental health problems.

[0003] At present, the traditional psychological intervention methods mainly include the following: (1) Psychological counseling and treatment: Professional counselors or psychotherapists provide one-on-one or group psychological intervention, mainly using cognitive behavioral therapy (CBT), humanistic therapy, psychoanalytic therapy and other methods to help individuals identify and deal with emotional problems. The advantage of this method is that it can provide in-depth emotional analysis and support based on the patient's specific situation, helping patients to develop self-healing abilities by exploring the roots of emotions and cognitive patterns. However, it also has some disadvantages, such as high time and economic costs, and relatively scarce psychological counseling resources.

[0004] (2) Drug treatment: For more serious mental health problems, such as depression, anxiety, bipolar disorder, etc., clinicians will prescribe antidepressants, antianxiety drugs and other drugs to help patients regulate their emotions and alleviate symptoms. Drug treatment is usually used in conjunction with other interventions to achieve the best results. The advantage of this method is that for patients with more severe symptoms, drugs can quickly improve emotions and behaviors and relieve symptoms, but drugs may cause a series of side effects and some drugs are addictive, so they need to be used with caution and gradually discontinued.

[0005] (3) Brain stimulation therapy: physical stimulation is used to directly or indirectly affect specific areas of the brain through electric current or magnetic field, regulate brain nerve activity, and improve mood and behavior. The advantage of this method is that it is particularly effective for patients with severe depression. Usually, a few treatments can significantly improve mood. However, there are also some disadvantages, such as the possibility of causing short-term memory loss, confusion and other side effects, and some patients may feel uncomfortable.

[0006] In summary, current psychological intervention methods mainly rely on the above methods. However, these mainstream methods have a significant problem, that is, patients with mental illness are often unable to perceive themselves in the early stages of the disease, which makes it difficult for medical staff to conduct preventive treatment in a timely manner and remind patients to seek help from professional psychological treatment institutions in a timely manner.

[0007] With the rapid development of artificial intelligence and computer vision technology, facial emotion recognition technology has been widely used in many fields, including smart homes, mental health monitoring, education, customer service, and human-computer interaction. However, although existing facial emotion recognition technology shows good accuracy in laboratory environments or controlled environments, the recognition effect is often greatly reduced in real, dynamic wild environments. Human emotions are not fixed, but dynamically fluctuating. In practical applications, an individual's emotions may change rapidly in a short period of time, and existing emotion recognition technologies are mostly based on static images or limited time series data, which makes it difficult to capture the entire process of emotional changes in a timely manner.

[0008] Most existing emotion recognition methods rely on only a single modality, namely facial images, and ignore the multimodal expression of emotions. However, the expression of emotions is multidimensional, and the recognition of a single modality may not fully reflect the true emotional state of an individual, resulting in certain errors in the results of emotion recognition. Summary of the invention

[0009] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a multimodal emotion recognition method and a psychological intervention system.

[0010] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0011] In a first aspect, the present invention provides a multimodal emotion recognition method, comprising: Collecting video clips of target facial features, and obtaining continuous frame image information based on the video clips; Perform face target detection on continuous frame image information and determine the face position in the image, standardize and normalize the detected face, and obtain a sequence of face feature points corresponding to the continuous frame image information; Extract key frames based on the facial feature point sequence and perform data enhancement to obtain the first feature vector; Extracting spatiotemporal features from continuous frame image information, then weighting the extracted features to output a second feature vector; The first feature vector and the second feature vector are jointly processed by an expression and feature point joint module to obtain an image feature vector; Collect the original audio signal of the target person, and extract the audio feature vector according to the original audio signal; The image feature vector and the audio feature vector are fused through an audio-visual tensor product fusion module to obtain a final fused feature vector, which is input into a pre-trained multimodal fusion emotion recognition network model based on the Transformer architecture to output the emotion recognition result of the target person.

[0012] Furthermore, the video clips for collecting target facial features, and obtaining continuous frame image information according to the video clips, include: The video clips of the target facial features are collected by the camera device, and the frame sequence is intercepted at a preset number of frames per second for each video clip. The image information of the frame sequence is processed by an error handling mechanism to obtain the processed continuous frame image information.

[0013] Furthermore, the adopting of the error handling mechanism to process the image information of the frame sequence to obtain processed continuous frame image information includes: Performing a time check on the image information to determine whether there is a frame loss; If there is frame loss, linear interpolation or motion prediction algorithm is used to estimate and reconstruct the lost frame image according to the time and motion characteristics of adjacent frame images; The Laplace operator is used to detect blurriness, and the blurred frames are compensated by interpolation of previous and next frames to obtain preliminary continuous frame image information; Using a pinhole vision model, distortion correction is performed on each image in preliminary continuous frame image information; Use the convolutional neural network model to denoise the distortion-corrected image; Perform color correction on the denoised image using a color correction algorithm; The color-corrected image is enhanced using the histogram equalization method; The enhanced image is subjected to error detection and recognition through machine learning methods, and the corrected error information existing in the image is repaired to obtain the processed continuous frame image information.

[0014] Furthermore, the method of performing face target detection on the continuous frame image information and determining the face position in the image, and performing standardization and normalization processing on the detected face to obtain a sequence of face feature points corresponding to the continuous frame image information includes: Use the YOLO algorithm to detect the face position and obtain the position of the target face; Generate a bounding box for each detected face using an object detection algorithm based on the location of the target face; The image of the generated bounding box is scaled using a bilinear interpolation algorithm and normalized to the target pixel to obtain a sequence of facial feature points corresponding to the continuous frame image information.

[0015] Furthermore, the step of extracting spatiotemporal features from the continuous frame image information, and then weighting the extracted features to output a second feature vector includes: The 3D convolution block is used to extract the spatiotemporal features of the continuous frame image information to obtain the spatial features, and the spatial attention mechanism is used to weight the obtained spatial features; Use the temporal attention mechanism to weight the weighted spatial features to obtain the spatiotemporal fusion feature vector; Global pooling is used to map the spatiotemporal fusion features into a fixed-length vector required by the model to obtain the second feature vector.

[0016] Furthermore, extracting the audio feature vector according to the original audio signal includes: The original audio signal is pre-emphasized using a first-order FIR high-pass digital filter; Using a sliding window algorithm to divide the pre-emphasized audio signal into audio segments of preset time; Use the Hamming window algorithm to window the audio clip; Calculate the energy of the windowed audio clips, remove frames with energy below the threshold, and obtain the processed audio signal; The preprocessed audio signal is converted into the frequency domain using fast Fourier transform to obtain a complex spectrum; The complex spectrum is divided into multiple Mel frequency bands using a Mel filter bank; Taking the logarithm of the Mel frequency band output by the Mel filter obtains the logarithmic Mel energy spectrum; The logarithmic Mel energy spectrum is transformed into Mel cepstral coefficient audio features as the audio feature vector using discrete cosine transform.

[0017] Furthermore, the image feature vector and the audio feature vector are fused by an audio-visual tensor product fusion module to obtain a final fused feature vector, including: Using a feature concatenation algorithm to concatenate the image feature vector and the audio feature vector in a feature dimension to form a vector pair; The elements of each dimension of the vector pair are interacted pair by pair through tensor product operation to calculate the fused joint features: Linearly weight the fused joint features; The linearly weighted features are passed through the output layer to generate the final fused feature vector.

[0018] Furthermore, the training process of the multimodal fusion emotion recognition network model based on the Transformer architecture includes: Construct a multimodal fusion emotion recognition model based on deep spatiotemporal network and spatiotemporal attention mechanism; According to the training set, it includes: using the public emotion calculation dataset as the basic data source; Perform data enhancement preprocessing on the basic data source to obtain the training set; During the training process, the spatial attention network in the spatiotemporal attention mechanism is used to weight the key areas in the facial images in the training set so that the deep spatiotemporal network can learn facial expression features, and the temporal attention mechanism in the spatiotemporal attention mechanism is used to dynamically adjust the weights of each time so that the deep spatiotemporal network can obtain different weights in time sequence; Use the label smoothing module to convert labels from hard-coded to smooth probability distribution; The deep spatiotemporal network is optimized using a stochastic gradient descent optimizer, and the learning rate range is set to perform model training to obtain a trained multimodal fusion emotion recognition network model based on the Transformer architecture.

[0019] In a second aspect, the present invention further provides a psychological intervention system, comprising: An acquisition module, used to acquire an emotion recognition result of a target person based on the multimodal emotion recognition method of the first aspect; A virtual reality module, which is used to develop and build virtual scenes based on the Unreal Engine game engine and scene editor, including multiple virtual objects and tasks related to alleviating negative emotions, as well as multiple virtual NPC characters; The psychological intervention task module is used to generate task objectives and feedback mechanisms based on cognitive behavioral therapy, mindfulness intervention, animal-assisted therapy, and emotional feedback theory, transmit the task objectives and feedback mechanisms to the virtual reality module, and dynamically adjust the task objectives based on the emotion recognition results of the target person; it is also used to automatically or manually adjust the task difficulty and task type according to the psychological state of the target person and the emotional feedback information of the target person.

[0020] Furthermore, the virtual reality module includes an NPC guidance unit, which includes multiple virtual NPC characters that intelligently interact based on the emotion recognition results and psychological needs of the target person. The virtual NPC characters dynamically adjust the actions, dialogues or expressions of task guidance, emotional support and instant feedback according to the instructions of the virtual reality module.

[0021] The beneficial effects achieved by the present invention are: First, by fusing expressions with feature points and combining them with an advanced cross-attention mechanism, the accuracy of the emotion recognition system in complex environments is effectively improved. This module is able to maintain efficient and stable emotion recognition capabilities in a changing dynamic environment. By using the spatiotemporal attention mechanism to extract feature image features, emotion recognition is not limited to a single static moment, but can track and capture dynamic changes in emotions in real time. By analyzing the process of emotional fluctuations, the system's ability to respond to emotional dynamics is enhanced.

[0022] Secondly, the present invention is suitable for users with various mental health problems such as emotional disorders, anxiety, depression or stress. The system can use virtual reality technology and multimodal emotion recognition technology to provide users with a safe and immersive virtual environment, while providing personalized psychological intervention to help users effectively relieve negative emotions and improve their mental health level. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is the flow chart of the main control module of the psychological intervention system; Figure 2 It is a flow chart of the multimodal emotion recognition module of the psychological intervention system; Figure 3 It is a schematic diagram of the user module of the psychological intervention system; Figure 4 It is a schematic diagram of the data transmission module of the psychological intervention system; Figure 5 Schematic diagram of the main control module of the psychological intervention system; Figure 6 This is a model structure diagram of the multimodal fusion emotion recognition network model based on the Transformer architecture. DETAILED DESCRIPTION

[0024] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0025] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0026] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0027] Embodiment 1: This embodiment introduces a multimodal emotion recognition method, including: Collecting video clips of target facial features, and obtaining continuous frame image information based on the video clips; Perform face target detection on continuous frame image information and determine the face position in the image, standardize and normalize the detected face, and obtain a sequence of face feature points corresponding to the continuous frame image information; Extract key frames based on the facial feature point sequence and perform data enhancement to obtain the first feature vector; Extracting spatiotemporal features from continuous frame image information, then weighting the extracted features to output a second feature vector; The first feature vector and the second feature vector are jointly processed by an expression and feature point joint module to obtain an image feature vector; Collect the original audio signal of the target person, and extract the audio feature vector according to the original audio signal; The image feature vector and the audio feature vector are fused through an audio-visual tensor product fusion module to obtain a final fused feature vector, which is input into a pre-trained multimodal fusion emotion recognition network model based on the Transformer architecture to output the emotion recognition result of the target person.

[0028] The video clips for collecting target facial features, and obtaining continuous frame image information according to the video clips, include: The video clips of the target facial features are collected by the camera device, and the frame sequence is intercepted at a preset number of frames per second for each video clip. The image information of the frame sequence is processed by an error handling mechanism to obtain the processed continuous frame image information.

[0029] The process of using an error handling mechanism to process the image information of the frame sequence to obtain processed continuous frame image information includes: Performing a time check on the image information to determine whether there is a frame loss; If there is frame loss, linear interpolation or motion prediction algorithm is used to estimate and reconstruct the lost frame image according to the time and motion characteristics of adjacent frame images; The Laplace operator is used to detect blurriness, and the blurred frames are compensated by interpolation of previous and next frames to obtain preliminary continuous frame image information; Using a pinhole vision model, distortion correction is performed on each image in preliminary continuous frame image information; Use the convolutional neural network model to denoise the distortion-corrected image; Perform color correction on the denoised image using a color correction algorithm; The color-corrected image is enhanced using the histogram equalization method; The enhanced image is subjected to error detection and recognition through machine learning methods, and the corrected error information existing in the image is repaired to obtain the processed continuous frame image information.

[0030] The method of performing face target detection on the continuous frame image information and determining the face position in the image, and performing standardization and normalization processing on the detected face, includes: Use the YOLO algorithm to detect the face position and obtain the position of the target face; Generate a bounding box for each detected face using an object detection algorithm based on the location of the target face; The image of the generated bounding box is scaled using a bilinear interpolation algorithm and normalized to the target pixel to obtain a sequence of facial feature points corresponding to the continuous frame image information.

[0031] The step of extracting spatiotemporal features from continuous frame image information, then weighting the extracted features, and outputting an image feature vector includes: The 3D convolution block is used to extract the spatiotemporal features of the continuous frame image information to obtain the spatial features, and the spatial attention mechanism is used to weight the obtained spatial features; Use the temporal attention mechanism to weight the weighted spatial features to obtain the spatiotemporal fusion feature vector; Global pooling is used to map the spatiotemporal fusion features into a fixed-length vector required by the model to obtain the second feature vector.

[0032] The step of extracting audio feature data according to the original audio signal comprises: The original audio signal is pre-emphasized using a first-order FIR high-pass digital filter; Using a sliding window algorithm to divide the pre-emphasized audio signal into audio segments of preset time; Use the Hamming window algorithm to window the audio clip; Calculate the energy of the windowed audio clips, remove frames with energy below the threshold, and obtain the processed audio signal; The preprocessed audio signal is converted into the frequency domain using fast Fourier transform to obtain a complex spectrum; The complex spectrum is divided into multiple Mel frequency bands using a Mel filter bank; Taking the logarithm of the Mel frequency band output by the Mel filter obtains the logarithmic Mel energy spectrum; The logarithmic Mel energy spectrum is transformed into Mel cepstral coefficient audio features as the audio feature data using discrete cosine transform.

[0033] The step of fusing the image feature vector and the audio feature vector through an audio-visual tensor product fusion module to obtain a final fused feature vector includes: Using a feature concatenation algorithm to concatenate the image feature vector and the audio feature vector in a feature dimension to form a vector pair; The elements of each dimension of the vector pair are interacted pair by pair through tensor product operation to calculate the fused joint features: Linearly weight the fused joint features; The linearly weighted features are passed through the output layer to generate the final fused feature vector. Furthermore, the training process of the multimodal fusion emotion recognition network model based on the Transformer architecture includes: Construct a multimodal fusion emotion recognition model based on deep spatiotemporal network and spatiotemporal attention mechanism; According to the training set, it includes: using the public emotion calculation dataset as the basic data source; Perform data enhancement preprocessing on the basic data source to obtain the training set; During the training process, the spatial attention network in the spatiotemporal attention mechanism is used to weight the key areas in the facial images in the training set so that the deep spatiotemporal network can learn facial expression features, and the temporal attention mechanism in the spatiotemporal attention mechanism is used to dynamically adjust the weights of each time so that the deep spatiotemporal network can obtain different weights in time sequence; The label smoothing module is used to convert the labels from hard-coded to smooth probability distribution to improve the generalization performance and accuracy of the model in the expression classification task; The deep spatiotemporal network is optimized using a stochastic gradient descent optimizer, and the learning rate range is set to 10^−3 to 10^−4 for model training to obtain a trained multimodal fusion emotion recognition network model based on the Transformer architecture.

[0034] It also includes: when the user does not actively select the game difficulty, dynamically adjusting the game difficulty through the recognized emotional results; the acquisition of the game difficulty includes: pre-setting the difficulty level division of seven emotions: happy, sad, angry, terrified, shocked, disgusted, peaceful and neutral.

[0035] Dynamically adjusting the game difficulty through the recognized emotion results, including: dynamically adjusting the game difficulty using a pre-trained dynamic difficulty adjustment model based on the recognized emotion results and in combination with individual characteristics of the user, wherein the individual characteristics include the user's age, gender, game experience, health status, and psychological state; The dynamic difficulty adjustment model uses technologies based on reinforcement learning and other techniques to determine the game difficulty that best suits the user through continuous trial and error and feedback.

[0036] Embodiment 2 is based on the same inventive concept as Embodiment 1. This embodiment introduces a psychological intervention system, including: An acquisition module, used to acquire an emotion recognition result of a target person using the multimodal emotion recognition method of Example 1; A virtual reality module, which is used to develop and build virtual scenes based on the Unreal Engine game engine and scene editor, including multiple virtual objects and tasks related to alleviating negative emotions, as well as multiple virtual NPC characters; The psychological intervention task module is used to generate task objectives and feedback mechanisms based on cognitive behavioral therapy, mindfulness intervention, animal-assisted therapy, and emotional feedback theory, transmit the task objectives and feedback mechanisms to the virtual reality module, and dynamically adjust the task objectives based on the emotion recognition results of the target person; it is also used to automatically or manually adjust the task difficulty and task type according to the psychological state of the target person and the emotional feedback information of the target person.

[0037] The virtual reality module includes an NPC guidance unit, which includes multiple virtual NPC characters that intelligently interact based on the emotion recognition results and psychological needs of the target person. The virtual NPC characters dynamically adjust the actions, dialogues or expressions of task guidance, emotional support and instant feedback according to the instructions of the virtual reality module.

[0038] Example 3 is based on the same inventive concept as Example 2. Figure 1 As shown, this embodiment introduces a psychological intervention method based on multimodal emotion recognition, including: Based on the system of Example 2, the user selects a display device, loads the initial game interface, selects the corresponding character, then starts the game and loads the game map and scene screen, enters the game level and interacts according to the instructions of the NPC. During the interaction, the recognition method in Example 1 is used to perform emotion recognition, and the computer dynamically adjusts the game difficulty and NPC guidance, and displays the data analysis results of the emotion recognition result set.

[0039] Example 4 is based on the same inventive concept as Example 1. Figure 1 ,picture, Figure 3 , Figure 4 , Figure 5 and Figure 6 As shown, this embodiment introduces a psychological intervention system, which can use virtual reality technology and multimodal emotion recognition technology to dynamically adjust the virtual reality game content to effectively alleviate the user's negative emotions and improve their mental health. At the same time, it can monitor and identify the user's emotional state in real time in a non-invasive manner, give feedback and suggestions, and promote the user's mental health recovery. The system includes: The virtual reality scene construction module is developed based on Unreal Engine and is a virtual scene built by the inventor himself. The scene includes multiple virtual objects and tasks related to relieving negative emotions, as well as multiple virtual NPC characters. The module is also used to select or personalize virtual objects, tasks and scenes suitable for users according to the user's emotional state, and adjust the appearance, personality and behavior of virtual NPC characters. The module uses the game engine and scene editor of Unreal Engine. The specific principles are as follows: Game Engine: Unreal Engine is a powerful cross-platform game development tool that provides a range of features, such as graphics rendering, physics simulation, blueprint visual scripting system, AI behavior tree, animation system, etc. It is widely used in the development of 3D games, virtual reality (VR), augmented reality (AR) and other interactive experiences. Unreal Engine's game engine is component-based. It abstracts each object in the game (such as characters, objects, lights, etc.) into a game object (GameObject). Each game object can be attached with multiple components (Component) to define the properties and behaviors of the game object, such as position, rotation, scale, collider, renderer, animation controller, etc. The relationship between game objects and components can be described by the following formula: .

[0040] Scene Editor: Unreal Engine provides a visual scene editor for creating and arranging game objects in the game, as well as setting the game environment and parameters. The scene editor consists of the following parts: Hierarchy View: Used to display and manage all game objects in the scene, as well as the parent-child relationships between them.

[0041] Inspector View: Used to display and edit the properties and components of the selected game object.

[0042] Scene View: Used to view and manipulate game objects in the scene in three-dimensional space, as well as set the camera's perspective and position.

[0043] Game View: Used to preview and run the game, as well as view the game's output and debugging information.

[0044] Project View: Used to display and manage all resources in the project, such as models, textures, sound effects, scripts, etc.

[0045] Console View: Used to display and filter game logs, warnings, and error messages.

[0046] The acquisition module is used to use the camera to capture the video clips of the user's facial features in real time, use the error handling mechanism to process the image information of the frame sequence, and obtain the complete continuous frame image information; use the recording device to obtain the user's audio information in real time; Psychological intervention task design module: The psychological intervention task design module is the core part of the whole psychological intervention method. Its goal is to gradually guide users to complete psychologically meaningful activities through a scientifically designed task system, thereby alleviating negative emotions and enhancing self-worth. This module transmits the task objectives and feedback mechanism to the main control module, and dynamically adjusts them in combination with the user's emotional state; in addition, this module is also used to automatically or manually adjust the task difficulty and task type according to the user's psychological state and emotional feedback to optimize the user's experience and intervention effect. This module is based on cognitive behavioral therapy, mindfulness intervention, emotional feedback theory, and animal-assisted therapy (AAT), and strives to achieve significant psychological counseling effects through gamified intervention tasks. The specific design principles are as follows: Cognitive behavioral therapy: By designing kind tasks, such as "rescue a fawn caught in a trap", "help a fallen chick return to its nest", and "rescue a butterfly trapped in a spider web to regain its freedom", users are encouraged to reshape their positive thinking patterns in the tasks and gradually build up their confidence and sense of self-worth. In the process of completing the tasks, users need to overcome obstacles, improve their problem-solving skills, and gain a sense of accomplishment through continuous efforts.

[0047] Mindfulness intervention: Through soothing background music, soft scene design and slow task rhythm in the task, users are guided to focus on the current activities and reduce their attention to external pressure. For example, the task "help the fallen chicks return to the nest" guides users to slow down, focus on the actions in front of them, and experience the peace of mind and inner focus in the process through meticulous operations, such as gently catching the chicks and finding the correct flight route. The task rhythm is neither fast nor slow, giving users enough time to complete the task, so as to better relax their body and mind and reduce anxiety. The module also designs emotional expression tasks, such as "writing a letter to a virtual character" or "selecting an icon that expresses current feelings", to help users release their inner emotions and focus on their inner feelings.

[0048] Animal Assisted Therapy (AAT): Incorporating the concept of animal-assisted therapy (AAT) into task design, users promote emotional connection and psychological recovery by helping virtual animals solve difficulties. The specific task description is as follows: Rescue a deer caught in a trap: In this task, the player will encounter a deer caught in a trap. The user needs to carefully observe the surrounding environment, identify and use appropriate methods to remove the trap. This task emphasizes the user's meticulous observation and action ability, helping the player to enhance self-efficacy through the process of analyzing and solving problems. After the task is completed, the deer is free and runs happily in the forest. This plot not only strengthens the player's emotional sense of achievement, but also stimulates positive emotional responses through the behavior of helping the deer, further promoting psychological recovery and positive emotional experience.

[0049] Help the fallen chicks return to the nest: This task requires the user to help the fallen chicks return to the high nest through clever means (such as building a small bridge or finding a suitable branch). In the process, the user must not only pay attention to the meticulousness and thoughtfulness of the action, but also ensure that the bird returns to the nest safely and without hindrance. Through this task, the player can feel the fragility of life and the sense of responsibility for others (or animals), while experiencing the warmth and satisfaction of helping fragile life regain freedom. Psychologically, the player can get emotional relief and positive feedback from the heart through this positive behavior and emotional connection.

[0050] Rescue a small butterfly trapped in a spider web and restore its freedom: This mission takes place in an area full of spider webs. Players need to use extremely delicate skills to rescue a small butterfly trapped in a spider web. The player's task is to carefully untie the butterfly and ensure its freedom. At the moment when the butterfly successfully flies into the blue sky, the player will feel a strong emotional release and psychological comfort. This mission not only relieves the player's anxiety by helping fragile lives, but also brings a deep sense of inner peace and pleasure through emotional expression and interaction, thereby effectively promoting the player's mental health.

[0051] These tasks help users relieve negative emotions through animal interaction and care, making users feel that their actions can bring about positive changes, enhancing their self-confidence and sense of emotional connection.

[0052] Emotional feedback theory: The real-time emotion recognition system monitors the user's emotional state and dynamically adjusts the task content according to the emotion recognition results. When the user is in a low mood, the system will reduce the difficulty of the task or increase encouraging feedback and provide positive feedback to the user through the character's voice, visual scene changes and reward mechanism to relieve their anxiety and enhance emotional stability; for example, in the task of "rescuing the deer caught in the trap", if the system recognizes that the player's emotion is anxious or nervous, it will reduce the complexity of the task by providing clear rescue steps or extending the task time, so that the player can complete the task smoothly without too much pressure. In addition, the system can play warm sound effects and provide emotional comfort through the character's voice feedback. When the user's emotions tend to be stable, the challenge of the task is appropriately increased, and more challenging elements are added to maintain the attractiveness of the task and the user's sense of participation. By evaluating the challenge of the task and the user's coping ability in real time, the task is gradually upgraded to help users improve their confidence. Progressive task difficulty design: Task design follows a progressive strategy from easy to difficult, building user confidence through simple initial tasks and improving their coping ability through gradually upgraded task challenges, ensuring that intervention tasks are always within the user's acceptable range and avoiding boredom or frustration caused by tasks that are too simple or difficult.

[0053] The facial expression recognition module is used to capture the user's facial expression from the user's complete continuous frame image information using a deep spatiotemporal network; this module utilizes the facial expression capture algorithm, facial expression feature extraction algorithm and facial expression key frame extraction algorithm. The specific principles are as follows: Facial expression capture algorithm: The facial expression capture algorithm is an advanced computer vision algorithm that can locate and extract the face area from the user's complete continuous frame image information, avoid being affected by interference information such as background information unrelated to the face when extracting expression features, and improve the distinction of expression features. The algorithm captures the user's facial features through an ordinary RGB camera and can provide accurate emotional feedback without dedicated hardware, which is crucial for psychological intervention systems. The core of the algorithm is to quickly process large amounts of data and accurately identify key facial features to achieve dynamic emotional tracking.

[0054] (1) Face detector: Face detection is the first step in expression capture, and its purpose is to locate the facial area in the input image. The detector is completed through a deep learning model. The model has been trained with a large number of facial images and can accurately identify facial contours. The detection result is usually the coordinate position of multiple facial frames. The probability distribution map output by the detector is generated by the Softmax function, which normalizes the probability of each pixel point to indicate the possibility that it belongs to a face. The Softmax function formula is as follows: ; in, P ( x) represents the probability that a pixel belongs to a face, x i is the original prediction value of the model for the pixel. The denominator is the sum of the predicted values ​​of all possible categories after exponential operation, ensuring that the sum of all probabilities is 1. This process not only improves classification accuracy, but can also be used in multi-category scenarios. It is a common classification method in deep learning.

[0055] (2) Key point coordinate prediction: After successfully detecting the face, the next step is to predict the coordinates of facial key points, including the eyes, mouth, nose, eyebrows, and 68 key points at the outer contour of the face. These key points are the basis for analyzing facial expressions. This module uses a regression model to predict the coordinates of the key points. A simple linear regression model assumes that the output variable y and input variables There is a linear relationship between them, and the goal is to find a set of parameters To minimize the predicted value ( ) and the true value ( y ): ; in, is the intercept term, are model parameters, are input features, is the error term, which represents other influencing factors besides the input features. The key point prediction model in the algorithm not only needs to predict a single point, but also needs to handle the simultaneous prediction of multiple key points, so a multi-output regression model is used to enhance accuracy.

[0056] (3) Depth estimation: To accurately capture three-dimensional facial expressions, two-dimensional images are not enough, and the depth information of key points must be estimated. Through methods similar to stereo vision, the depth estimation algorithm can infer the position of each key point in three-dimensional space. The basis of depth estimation is the principle of triangulation, which calculates the depth through the distance between two cameras (baseline distance) and the parallax of corresponding points. The formula is as follows: ; in, Z is the depth value, f is the focal length of the camera, B is the baseline distance between the two cameras, d is the disparity of corresponding points on the image, that is, the horizontal position difference of the same object in the two camera images. This formula shows that disparity and depth are inversely proportional: the larger the disparity, the smaller the depth, and vice versa. In the algorithm, even if only one camera is used, the algorithm can simulate stereo vision through machine learning and estimate the depth information of a single image.

[0057] Facial expression feature extraction algorithm: The key step of facial expression feature extraction is to extract effective features representing expressions from the facial area. This step is usually implemented through a convolutional neural network (CNN), which is a deep learning model that processes image data through a series of convolutional layers. The convolution operation is the basic operation of CNN, which slides a filter (or convolution kernel) on the input image and calculates the dot product between the filter and the local area of ​​the image to extract features.

[0058] The innovation of this algorithm is that it first uses 3D convolution blocks to extract the spatiotemporal features of the input expression image sequence, and then adds a spatiotemporal attention module after each 3D convolution block, and uses the spatial attention mechanism and the temporal attention mechanism to weight the obtained expression spatiotemporal features to obtain more accurate expression-related spatiotemporal features. In addition, in order to effectively fuse the multimodal information of expression images and facial feature points, the present invention designs a joint module to extract the spatiotemporal features from the expression image sequence ( ) and facial feature point sequence ( ) and configures an independent encoder for each feature type. In addition, the algorithm also introduces label smoothing technology to improve the generalization ability of the model by smoothing the labels, reduce the risk of overfitting, and thus improve the accuracy and robustness of the model. The specific formula is as follows: (1) 3D convolution operation: First, the input facial video is subjected to preliminary feature extraction through multiple 3D convolution layers to obtain spatiotemporal features. In a 3D convolutional neural network, a 3D convolution kernel is used to perform a convolution operation on a cube composed of multiple adjacent frames. The feature map in the current convolution layer is connected to multiple consecutive frames in the previous layer to capture motion information. i Layer j The position on the feature map ( x , y , z ) is expressed as , calculated by the following formula: ; in, represents the activation function; Q i Indicates the size of the 3D convolution kernel temporal dimension; P i Indicates i The size of the convolution kernel of the layer in the spatial dimension, R i Indicates i The stride of the convolution kernel of the layer in the time dimension; Indicates that the convolution kernel connected to the 𝑚th feature map of the previous layer is at position ( p ,q , the value of 𝑟).

[0059] (2) Spatial attention mechanism: The spatial attention mechanism can help the model highlight the important areas of facial expressions in the spatial dimension and enhance the perception of expression details. At the same time, the spatial attention mechanism can dynamically adjust the attention to different spatial positions to adapt to the changes in expression characteristics at different time points. It is calculated by the following formula: ; in, Represents a convolution operation with a 7×7 convolution kernel, and the concatenated feature map is processed through a convolution layer; this operation generates a spatial attention map , it is a A two-dimensional matrix of Represents the feature map obtained after the average pooling operation; Represents the feature map obtained after the maximum pooling operation.

[0060] Then, the spatial attention map Multiply it with the input feature map F to get the weighted feature map : ; in, represents element-wise multiplication. This operation enables the network to strengthen the key areas of facial expressions according to the spatial attention value and suppress the influence of unimportant areas.

[0061] (3) Temporal attention mechanism: Dynamic expressions contain rich temporal information. The temporal attention mechanism enables the network to dynamically focus on the importance of different positions in the sequence at each time step, which helps the model better understand and utilize temporal information. The temporal attention mechanism can improve the model's expressiveness and recognition accuracy by dynamically adjusting and weighting temporal information. It is calculated using the following formula: ; in, is the Sigmoid activation function, which maps the output to In the range of , it is used to express the importance weight of each time frame. and is the global information of each time frame, with size T ×1.

[0062] Then, the temporal attention map With spatial feature map Multiply to get the weighted feature map : ; in, This operation enables the network to pay more attention to the key moments of expression changes, thereby more effectively recognizing dynamic expressions.

[0063] (4) Joint module of expression and feature points: In multimodal emotion recognition, expression image sequences and facial feature point sequences provide different emotional clues. The joint feature fusion module combines the features of these two information sources to capture emotional expressions more comprehensively.

[0064] like Figure 2 and Figure 6 As shown in the figure, the facial feature point sequence is output to the 2D convolution network to obtain a feature vector (that is, the first feature vector), which is then output to the expression and feature point joint module; the 3D convolution block is used to extract the spatiotemporal features of the continuous frame image information to obtain the spatial features, and the spatial attention mechanism is used to weight the obtained spatial features; the time attention mechanism is used to weight the weighted spatial features to obtain the spatiotemporal fusion feature vector; the spatiotemporal fusion features are mapped into a fixed-length vector required by the model using global pooling to obtain the second feature vector; the second feature vector obtained after the spatiotemporal extraction and the feature vector (the first feature vector) of the facial feature point sequence are input to the expression and feature point joint module for joint operation, specifically: First, through the concatenation operation, the features of the expression image sequence are and the features of facial feature point sequences Perform concatenation to obtain the joint feature representation: ; in Represents a serial operation, Represents the features of the expression image sequence, Represents the features of a sequence of facial feature points.

[0065] In order to effectively process these joint features, three independent encoders are configured for each feature type. Each encoder contains a multi-head self-attention mechanism and a fully connected feedforward network. Residual connections and layer normalization operations are applied between these layers. Through the self-attention mechanism, the model can effectively focus on key features from different information sources and enhance the representation of important information in subsequent processing. The specific formula is as follows: .

[0066] (5) Label smoothing: Label smoothing is a technique to improve the generalization ability of a model. It is often used in classification tasks to reduce the model's excessive focus on a single category. The core idea is to transform the original one-hot label encoding into a more flexible probability distribution so that the model can also pay attention to information from other categories, thereby reducing sensitivity to noisy data and enhancing generalization ability. This can be obtained from the following formula: First, in the multi-classification task, for the original one-hot encoded label y , label smoothing will convert it from hard labels to soft labels: ; in, N is the total number of expression categories, and the original true label is y , the smoothed label is , the label smoothing parameter is .

[0067] After label smoothing is applied, the cross entropy loss function is redefined to enable the model to learn based on flexible labels as follows: ; is the label probability after smoothing, is the model's predicted probability for the i-th category.

[0068] Facial expression keyframe extraction algorithm: In a video or image sequence, a keyframe is a frame with high discrimination and the most significant expression intensity, which is used to represent the overall expression sequence. For facial expression sequences, keyframes often appear at the moment when the expression intensity reaches its peak. Generally, in a sequence where the expression intensity changes from "neutral-rising-peak", the last frame is the keyframe; while in a sequence where the expression intensity changes from "neutral-rising-peak-falling-neutral", the keyframe position is not fixed. Therefore, keyframe extraction should be adjusted according to the characteristics of different expression sequences to ensure that the peak of emotions is effectively captured. First, the vector of the feature point needs to be calculated, which can be obtained by the following formula: ; Where n represents the number of facial feature points in the t-th frame image, represents the facial feature points of the expression image in the tth frame of the expression sequence, Represents the coordinates of the kth feature point in the tth frame image.

[0069] In order to reduce the displacement error of feature points between adjacent frames, the feature points are scaled normalized. Assume that in the t-th frame image, the coordinates of the feature point corresponding to the nose are ( ), the feature point normalization operation is as follows: ; in, ( ) represents the standard deviation of all feature point coordinates of the 𝑡th frame image; the normalized feature point vector for: ; Assuming that the first frame of the expression sequence is a neutral expression, the relative displacement difference between the corresponding feature points of the tth frame image and the first frame image is : ; In an expression sequence, calculate The frame image corresponding to the maximum value is the key frame required by the present invention.

[0070] Audio feature extraction module: The audio feature extraction module is used to extract emotion-related audio features from the user's original audio signal, and generate high-quality audio feature vectors for emotion recognition through a series of preprocessing and feature extraction steps. This module includes audio signal preprocessing and audio feature vector generation. The specific principles are as follows: Audio signal preprocessing: Audio signal preprocessing is a preliminary processing step for raw audio data, with the goal of improving signal quality and removing noise so that audio features can be extracted more effectively. It mainly includes steps such as pre-emphasis, framing, windowing, and energy screening.

[0071] (1) Audio pre-emphasis: Use a first-order FIR high-pass digital filter to pre-emphasize the original audio signal, enhance the high-frequency components, and make the audio signal clearer in the subsequent processing stage. The specific formula is as follows: ; (2) Framing: The pre-emphasized audio signal is divided into short segments of 30 milliseconds using a sliding window algorithm to capture short-term speech features. Each segment overlaps with each other to facilitate continuous signal processing. Each frame contains N sampling points (assuming the sampling rate is 16 kHz, then N = 480). The frame formula is expressed as: ; in, is the audio signal of the kth frame, H is the frame shift (for example, H=N / 2 when the frame overlap is 50%); (3) Audio windowing: By multiplying the audio signal, the limited-length window function can be shifted or converted. The signal after windowing is: ; In order to effectively retain the information content of the speech signal during its change process to the greatest extent possible, a sliding window is used. The speech sampling sequence formed after windowing is: ; (4) Energy screening: Calculate the energy of the windowed audio clips and remove frames with energy below the threshold to remove low-energy or noisy audio clips and improve the accuracy of speech features. The specific formula is as follows: ; if If the frame value is lower than the set threshold, the frame will be discarded.

[0072] Frequency domain feature extraction: Frequency domain feature extraction is the process of converting audio signals from time domain to frequency domain in order to analyze their spectral characteristics. It mainly includes fast Fourier transform and Mel frequency cepstral coefficients. The specific steps are as follows: 1) Fast Fourier Transform: Perform fast Fourier transform on the preprocessed audio clip to convert the time domain signal into a frequency domain complex spectrum to obtain the amplitude and phase information of each frequency component. The specific formula is as follows: ; in, is the time domain signal, is the corresponding frequency domain representation; 2) Mel frequency cepstral coefficients: Apply discrete cosine transform (DCT) to the logarithmic Mel energy spectrum to obtain Mel cepstral coefficients. These coefficients represent the acoustic characteristics of the audio and are important input features for sentiment analysis. The specific formula is as follows: ; in, is the energy of the Mel frequency band, n is the index of the MFCC coefficient, and K is the number of Mel frequency bands.

[0073] The audio-visual tensor product fusion module is used to fuse the above image feature vectors and audio feature vectors, and fuse these features through a linear weighting mechanism to improve the accuracy and stability of emotion recognition. This module accepts the feature vectors transmitted from the image feature extractor and the audio feature extractor, and then performs feature fusion through the weighted average method. The specific method is as follows: Specifically, in the image features and audio features Based on , the tensor product operation is used to fuse the visual and auditory features. The fused joint features Calculated by the following formula: ; in, represents the tensor product operation, Represents a concatenation operation.

[0074] For example, for the joint features of the i-th multimodal data pair, the calculation formula is as follows: ; The multimodal emotion recognition module is used to receive the user's image information and audio information transmitted by the main control module, use the linear support vector machine (SVM) to fuse the features to complete the user's emotion recognition, and then use the weighted probability value to judge the final emotion category and output the emotion recognition result. In order to fuse the image feature vector and the audio feature vector and improve the accuracy and stability of emotion recognition, the weighted average fusion method is adopted, which is defined as follows: ; in, is the emotion prediction probability value of the image or audio model for the input data x, and is the output of the softmax layer of the convolutional network. is the fusion weight of each model. The weight parameter 𝜔 is obtained by minimizing the loss function using least squares estimation on the validation set: .

[0075] The voice interaction module is used for voice interaction between the user and the system, receives the user's voice commands and feedback, and transmits the voice information to the main control module; The NPC guidance module includes multiple virtual NPC characters that interact intelligently based on the user's emotional state and psychological needs. According to the instructions of the main control module, it dynamically adjusts task guidance, emotional support and instant feedback. This module uses the animation system and voice system of Unreal Engine. The specific principles are as follows: Animation system: Unreal Engine provides a powerful animation system that can flexibly control the animation state of the character to provide emotional support during the interaction, including the following parts: Animation Asset: used to store the animation data of the character, such as walking, smiling, hugging and other actions. These animations can convey positive emotions such as warmth and support in virtual situations. Animations can be created using external tools, or edited and debugged within the engine.

[0076] Animation Blueprint: Used to manage the character's animation status and transition conditions, and control the switching of animations in different situations.

[0077] Animation State Machine: Used to define different animation states and transition conditions for a character. For example, the transition condition from "normal state" to "soothing action" can be dynamically triggered based on the player's emotional data.

[0078] Skeletal Mesh: Used for skeletal animation of characters to achieve natural movement performance and help characters vividly express gestures of care or support.

[0079] Voice system: The voice system in Unreal Engine is used to play and control the voice of the character, convey emotional support and feedback through sound, and specifically includes the following parts: Voice assets (Sound Cue): used to store the voice content of the character, such as greetings, encouragement, and comfort, to help players get emotional support during interaction.

[0080] Audio Source: The properties used to control voice playback, such as volume, pitch, and position, can be dynamically adjusted based on the user's state. For example, when the user is depressed, the system can lower the volume and play a soothing voice.

[0081] Audio Listener: used to receive and process voice data, adjust the voice effect according to the user's location and context, and make emotional comfort more immersive.

[0082] The main control module is used to control the operation of the entire system. It determines the user's psychological state and needs based on the user's emotion recognition results, voice information and individual characteristics, controls the changes in the virtual rehabilitation scene, controls the behavior of the NPC guidance module, controls the recording of the emotion data recording module, and controls the analysis of the data analysis module. This module uses the blueprint system and event system of Unreal Engine. The specific principles are as follows: Blueprint system: Unreal Engine provides a flexible blueprint system that supports a graphical programming interface for creating and executing the logic of game objects. The blueprint system implements logical control by dragging and connecting nodes, which is suitable for no-code or low-code development. It mainly includes the following parts: Blueprint Class: Used to store the logic of game objects, such as variables, functions, and nodes, which can implement the control logic of the entire system.

[0083] Blueprint Component: Attach a blueprint to a game object to give it the corresponding logical functions. For example, objects related to rehabilitation tasks in a character or scene can be attached with specific blueprint components to achieve dynamic control.

[0084] Blueprint Lifecycle: used to define the execution order and time points of the blueprint, such as initialization (Begin Play), update (Tick), destruction (Destroy), etc., to ensure that the entire psychological intervention system runs step by step.

[0085] Event system: Unreal Engine provides a unified event system for responding to and processing various events in the system, including user interaction, voice feedback, and context changes. The core parts of the event system are as follows: Event: used to indicate a specific situation that occurs in the system, such as a change in the user's gesture, a voice feedback trigger, etc. Each key emotional feedback or rehabilitation action in the system can trigger a specific event.

[0086] Event Source: A game object used to generate specific events, such as user input, virtual camera, etc.

[0087] Event Listener: used to receive and process specific events. For example, when the user completes a rehabilitation action, the event listener can receive information and call related feedback mechanisms.

[0088] Event Handler: used to define the specific processing logic of an event. For example, when a user completes a specific rehabilitation task, the system triggers voice feedback or NPC encouragement actions.

[0089] Data storage module: used to save user historical data, emotion recognition results and system feedback, and support subsequent personalized adjustment and analysis; It also includes: a data analysis module for analyzing users, dynamically adjusting game difficulty, and providing personalized suggestions; The data analysis module includes a data processor and a data display. The data processor is used to analyze the user's emotion recognition data and behavior feedback, use data mining and machine learning methods to evaluate the user's psychological state changes and emotional fluctuations, and generate corresponding analysis reports and suggestions. The data display is responsible for intuitively displaying the analysis results to the user in the form of charts or texts to facilitate understanding and further decision-making.

[0090] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0094] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multimodal emotion recognition method, characterized in that: include: Collecting video clips of target facial features, and obtaining continuous frame image information based on the video clips; Perform face target detection on continuous frame image information and determine the face position in the image, standardize and normalize the detected face, and obtain a sequence of face feature points corresponding to the continuous frame image information; Extract key frames based on the facial feature point sequence and perform data enhancement to obtain the first feature vector; Extracting spatiotemporal features from continuous frame image information, then weighting the extracted features to output a second feature vector; The first feature vector and the second feature vector are jointly processed by an expression and feature point joint module to obtain an image feature vector; Collect the original audio signal of the target person, and extract the audio feature vector according to the original audio signal; The image feature vector and the audio feature vector are fused through an audio-visual tensor product fusion module to obtain a final fused feature vector, which is input into a pre-trained multimodal fusion emotion recognition network model based on the Transformer architecture to output the emotion recognition result of the target person.

2. The multimodal emotion recognition method according to claim 1, characterized in that: The video clips for collecting target facial features, and obtaining continuous frame image information according to the video clips, include: The video clips of the target facial features are collected by the camera device, and the frame sequence is intercepted at a preset number of frames per second for each video clip. The image information of the frame sequence is processed by an error handling mechanism to obtain the processed continuous frame image information.

3. The multimodal emotion recognition method according to claim 2, characterized in that: The process of using an error handling mechanism to process the image information of the frame sequence to obtain processed continuous frame image information includes: Performing a time check on the image information to determine whether there is a frame loss; If there is frame loss, linear interpolation or motion prediction algorithm is used to estimate and reconstruct the lost frame image according to the time and motion characteristics of adjacent frame images; The Laplace operator is used to detect blurriness, and the blurred frames are compensated by interpolation of previous and next frames to obtain preliminary continuous frame image information; Using a pinhole vision model, distortion correction is performed on each image in preliminary continuous frame image information; Use the convolutional neural network model to denoise the distortion-corrected image; Perform color correction on the denoised image using a color correction algorithm; The color-corrected image is enhanced using the histogram equalization method; The enhanced image is subjected to error detection and recognition through machine learning methods, and the corrected error information existing in the image is repaired to obtain the processed continuous frame image information.

4. The multimodal emotion recognition method according to claim 1, characterized in that: The method of performing face target detection on the continuous frame image information and determining the face position in the image, and performing standardization and normalization processing on the detected face, includes: Use the YOLO algorithm to detect the face position and obtain the position of the target face; Generate a bounding box for each detected face using an object detection algorithm based on the location of the target face; The image of the generated bounding box is scaled using a bilinear interpolation algorithm and normalized to the target pixel to obtain a sequence of facial feature points corresponding to the continuous frame image information.

5. The multimodal emotion recognition method according to claim 1, characterized in that: The step of extracting spatiotemporal features from the continuous frame image information, and then weighting the extracted features to output a second feature vector includes: The 3D convolution block is used to extract the spatiotemporal features of the continuous frame image information to obtain the spatial features, and the spatial attention mechanism is used to weight the obtained spatial features; Use the temporal attention mechanism to weight the weighted spatial features to obtain the spatiotemporal fusion feature vector; Global pooling is used to map the spatiotemporal fusion features into a fixed-length vector required by the model to obtain the second feature vector.

6. The multimodal emotion recognition method according to claim 1, characterized in that: The step of extracting an audio feature vector according to the original audio signal comprises: The original audio signal is pre-emphasized using a first-order FIR high-pass digital filter; Using a sliding window algorithm to divide the pre-emphasized audio signal into audio segments of preset time; Use the Hamming window algorithm to window the audio clip; Calculate the energy of the windowed audio clips, remove frames with energy below the threshold, and obtain the processed audio signal; The preprocessed audio signal is converted into the frequency domain using fast Fourier transform to obtain a complex spectrum; The complex spectrum is divided into multiple Mel frequency bands using a Mel filter bank; Taking the logarithm of the Mel frequency band output by the Mel filter obtains the logarithmic Mel energy spectrum; The logarithmic Mel energy spectrum is transformed into Mel cepstral coefficient audio features as the audio feature vector using discrete cosine transform.

7. The multimodal emotion recognition method according to claim 1, characterized in that: The step of fusing the image feature vector and the audio feature vector through an audio-visual tensor product fusion module to obtain a final fused feature vector includes: Using a feature concatenation algorithm to concatenate the image feature vector and the audio feature vector in a feature dimension to form a vector pair; The elements of each dimension of the vector pair are interacted pair by pair through tensor product operation to calculate the fused joint features: Linearly weight the fused joint features; The linearly weighted features are passed through the output layer to generate the final fused feature vector.

8. The multimodal emotion recognition method according to claim 1, characterized in that: The training process of the multimodal fusion emotion recognition network model based on the Transformer architecture includes: Construct a multimodal fusion emotion recognition model based on deep spatiotemporal network and spatiotemporal attention mechanism; According to the training set, it includes: using the public emotion calculation dataset as the basic data source; Perform data enhancement preprocessing on the basic data source to obtain the training set; During the training process, the spatial attention network in the spatiotemporal attention mechanism is used to weight the key areas in the facial images in the training set so that the deep spatiotemporal network can learn facial expression features, and the temporal attention mechanism in the spatiotemporal attention mechanism is used to dynamically adjust the weights of each time so that the deep spatiotemporal network can obtain different weights in time sequence; Use the label smoothing module to convert labels from hard-coded to smooth probability distribution; The deep spatiotemporal network is optimized using a stochastic gradient descent optimizer, and the learning rate range is set to perform model training to obtain a trained multimodal fusion emotion recognition network model based on the Transformer architecture.

9. A psychological intervention system, characterized in that: include: An acquisition module, used to acquire an emotion recognition result of a target person based on the multimodal emotion recognition method according to any one of claims 1 to 8; A virtual reality module, which is used to develop and build virtual scenes based on the Unreal Engine game engine and scene editor, including multiple virtual objects and tasks related to alleviating negative emotions, as well as multiple virtual NPC characters; The psychological intervention task module is used to generate task objectives and feedback mechanisms based on cognitive behavioral therapy, mindfulness intervention, animal-assisted therapy, and emotional feedback theory, transmit the task objectives and feedback mechanisms to the virtual reality module, and dynamically adjust the task objectives based on the emotion recognition results of the target person; it is also used to automatically or manually adjust the task difficulty and task type according to the psychological state of the target person and the emotional feedback information of the target person.

10. The psychological intervention system according to claim 9, characterized in that: The virtual reality module includes an NPC guidance unit, which includes multiple virtual NPC characters that intelligently interact based on the emotion recognition results and psychological needs of the target person. The virtual NPC characters dynamically adjust the actions, dialogues or expressions of task guidance, emotional support and instant feedback according to the instructions of the virtual reality module.

Images