Psychological intervention methods, systems, equipment and media based on multimodal emotion recognition

Abstract

The present invention proposes a psychological intervention method, system, device and medium based on multimodal emotion recognition, the method comprising: preprocessing video data and voice data respectively; performing emotion recognition on the frame image corresponding to each time stamp of the preprocessed video data to obtain the image emotion recognition result corresponding to each time stamp; inputting the audio, the first task prompt word and the second task prompt word corresponding to each time stamp of the preprocessed audio data into the speech transcription emotion recognition integrated model to obtain the speech emotion recognition result corresponding to each time stamp; obtaining the final emotion recognition result corresponding to each time stamp according to the image emotion recognition result and the speech emotion recognition result; generating an emotion state recognition report according to the final emotion recognition result corresponding to each time stamp, so as to perform psychological intervention on the user according to the emotion state recognition report. The present invention improves the accuracy of emotion recognition and effectively improves the accuracy of psychological intervention.

Description

Technical Field

[0001] The present invention relates to the technical field of computers and psychology, and in particular to a psychological intervention method, system, device and medium based on multimodal emotion recognition. Background Art

[0002] With the prosperity and development of society and economy, people's living standards have been greatly improved. The existing psychological counseling method is generally to find a psychological counselor, vent the conflicts and accumulated emotions suppressed in the heart through talking with the counselor, and get the counselor's guidance, but finding a psychological counselor takes a lot of time and money, and many patients with mental illness have communication barriers, and are generally more defensive towards people.

[0003] The intelligent psychological robot consulting system is an innovative consulting tool that combines artificial intelligence technology with psychological knowledge. In the process of human-computer communication, emotions are the result of the comprehensive expression of multiple information. Single-mode emotion recognition is difficult to meet the actual needs due to its own limitations. Voice emotions are easily disturbed by noise, and expression recognition is easily affected by light or angle, which cannot achieve the ideal recognition effect, thus affecting the effect of intelligent psychological consultation. Summary of the invention

[0004] The present invention provides a psychological intervention method, system, device and medium based on multimodal emotion recognition, the main purpose of which is to improve the accuracy of emotion recognition and effectively improve the accuracy of psychological intervention.

[0005] In a first aspect, an embodiment of the present invention provides a psychological intervention method based on multimodal emotion recognition, comprising:

[0006] Acquire video data and voice data generated during the human-computer interaction process, and pre-process the video data and voice data respectively;

[0007] Performing emotion recognition on the frame image corresponding to each time stamp of the preprocessed video data to obtain the image emotion recognition result corresponding to each time stamp;

[0008] Input the audio, the first task prompt word and the second task prompt word corresponding to each timestamp of the preprocessed audio data into the speech transcription emotion recognition integrated model to obtain the speech emotion recognition result corresponding to each timestamp, wherein the speech transcription emotion recognition integrated model is obtained after training with samples, the first task prompt word is the transcribed audio text, and the second task prompt word is the recognized audio emotion;

[0009] According to the image emotion recognition result corresponding to each timestamp and the speech emotion recognition result corresponding to each timestamp, a final emotion recognition result corresponding to each timestamp is obtained;

[0010] An emotional state recognition report is generated according to the final emotion recognition result corresponding to each timestamp, so as to perform psychological intervention on the user according to the emotional state recognition report.

[0011] Furthermore, the speech transcription emotion recognition integrated model includes an encoder, a text embedder and a decoder of an enhanced language model. The audio, the first task prompt word and the second task prompt word corresponding to each timestamp of the preprocessed audio data are input into the speech transcription emotion recognition integrated model to obtain the speech emotion recognition result corresponding to each timestamp. The steps include:

[0012] Input the audio corresponding to each timestamp into the encoder to obtain an audio vector;

[0013] Inputting the first task prompt word and the second task prompt word into the text embedder respectively to obtain a first task vector and a second task vector;

[0014] Splicing the audio vector and the first task vector to obtain a first splicing vector, and splicing the first task vector and the second task vector to obtain a second splicing vector;

[0015] The first splicing vector and the second splicing vector are input into the decoder to obtain the transcription text corresponding to each timestamp and the speech emotion recognition result corresponding to each timestamp.

[0016] Further, the step of obtaining a final emotion recognition result corresponding to each timestamp according to the image emotion recognition result corresponding to each timestamp and the speech emotion recognition result corresponding to each timestamp comprises:

[0017] For each timestamp, if the image emotion recognition result and the voice emotion recognition result are consistent, the image emotion recognition result / the voice emotion recognition result is used as the final emotion recognition result;

[0018] If the image emotion recognition result is inconsistent with the speech emotion recognition result, the final emotion recognition results corresponding to multiple timestamps before the current timestamp are obtained, and the emotion with the highest frequency is taken as the final emotion recognition result.

[0019] Furthermore, the step of generating an emotion state recognition report according to the final emotion recognition result corresponding to each timestamp includes:

[0020] Get every round of dialogue during human-computer interaction;

[0021] According to the occurrence time of each round of conversation and the final emotion recognition result corresponding to each timestamp, the real-time emotion corresponding to each round of conversation is obtained;

[0022] Integrate the conversation content and the real-time emotion of each round of conversation into the third task prompt words corresponding to each round of conversation;

[0023] Input the third prompt word corresponding to each round of dialogue into the large language model to obtain the emotion category corresponding to each round of dialogue;

[0024] The proportion of each emotion category is counted to generate the emotion state recognition report.

[0025] Further, the image emotion recognition result and the voice emotion recognition result include fear, frustration, anxiety, anger, happiness, surprise, excitement, disgust, and stress;

[0026] The emotion categories include rich emotion, weak emotion, lack of emotion, and inappropriate emotion.

[0027] Furthermore, the step of performing emotion recognition on the frame image corresponding to each time stamp of the preprocessed video data to obtain the image emotion recognition result corresponding to each time stamp includes:

[0028] At least 3 timestamps are set per second, and frame images corresponding to each timestamp of the preprocessed video data are extracted;

[0029] The frame image corresponding to each timestamp is input into the visual language model to obtain the image emotion recognition result corresponding to each timestamp.

[0030] Furthermore, the steps of preprocessing the video data and the voice data respectively include:

[0031] Performing format conversion and denoising operations on the video data in sequence to obtain preprocessed video data;

[0032] Perform noise reduction and echo cancellation operations on the voice data to obtain pre-processed voice data.

[0033] In a second aspect, an embodiment of the present invention provides a psychological intervention system based on multimodal emotion recognition, comprising:

[0034] A preprocessing module, used to obtain video data and voice data generated during the human-computer interaction process, and preprocess the video data and voice data respectively;

[0035] An image recognition module is used to perform emotion recognition on the frame image corresponding to each time stamp of the preprocessed video data to obtain the image emotion recognition result corresponding to each time stamp;

[0036] A speech recognition module is used to input the audio, the first task prompt word and the second task prompt word corresponding to each timestamp of the preprocessed audio data into a speech transcription emotion recognition integrated model to obtain a speech emotion recognition result corresponding to each timestamp, wherein the speech transcription emotion recognition integrated model is obtained after training with samples, the first task prompt word is the transcribed audio text, and the second task prompt word is the recognized audio emotion;

[0037] A fusion module, used to obtain a final emotion recognition result corresponding to each timestamp according to the image emotion recognition result corresponding to each timestamp and the speech emotion recognition result corresponding to each timestamp;

[0038] The intervention module is used to generate an emotional state recognition report according to the final emotion recognition result corresponding to each timestamp, so as to perform psychological intervention on the user according to the emotional state recognition report.

[0039] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned psychological intervention method based on multimodal emotion recognition when executing the computer program.

[0040] In a fourth aspect, an embodiment of the present invention provides a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned psychological intervention method based on multimodal emotion recognition are implemented.

[0041] The present invention proposes a psychological intervention method, system, device and medium based on multimodal emotion recognition, which first pre-processes the video data and voice data generated in the human-computer interaction process; then performs emotion recognition on the frame image corresponding to each timestamp in the pre-processed video to obtain the image emotion recognition result corresponding to each timestamp; then inputs the audio, the first task prompt word and the second task prompt word corresponding to each timestamp into the speech transcription emotion recognition integrated model to obtain the speech emotion recognition result corresponding to each timestamp; finally, the image emotion recognition result and the speech emotion recognition result are fused to obtain the final emotion recognition result, and an emotion state recognition report is generated to perform psychological intervention on the user.

[0042] Since emotion recognition of a single modality may have deviations, the present invention performs emotion recognition based on video data and voice data respectively, and integrates the two emotion recognition results to obtain the final emotion recognition result, making the emotion recognition result more accurate; in addition, when traditional speech is transcribed into text and emotion recognition is performed, computational and storage redundancy is generated. The integrated speech transcription and emotion recognition model proposed in the present invention reuses the audio features extracted by the encoder. In this process, the audio encoder only needs to encode once, thereby greatly saving computational and storage costs and improving the real-time performance of the analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of a psychological intervention method based on multimodal emotion recognition provided by an embodiment of the present invention.

[0044] Figure 2 A structural diagram of a speech transcription and emotion recognition integrated model provided in an embodiment of the present invention.

[0045] Figure 3 A schematic diagram of the structure of a psychological intervention system based on multimodal emotion recognition provided by an embodiment of the present invention.

[0046] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention.

[0047] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0049] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0050] In the embodiments of the present application, at least one refers to one or more; multiple refers to two or more. In the description of the present application, words such as "first", "second", "third", etc. are only used to distinguish the purpose of description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0051] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include a particular feature, structure or characteristic described in conjunction with the embodiment. Thus, in this specification, the terms "include", "comprises", "has" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0052] Figure 1 A flowchart of a psychological intervention method based on multimodal emotion recognition provided by an embodiment of the present invention, the method comprising:

[0053] A psychological intervention method based on multimodal emotion recognition provided by an embodiment of the present invention can be applied in an intelligent psychological robot consulting system. When psychological intervention is needed, a corresponding instruction is sent to the intelligent psychological robot consulting system. After receiving the instruction, the intelligent psychological robot consulting system executes the psychological intervention method based on multimodal emotion recognition and outputs an emotional state recognition report.

[0054] S110, acquiring video data and voice data generated during the human-computer interaction process, and preprocessing the video data and the voice data respectively;

[0055] The embodiment of the present invention first obtains the video data and voice data generated during the human-computer interaction process. Specifically, during the interaction between the user and the intelligent psychological robot consulting system, the user generally communicates through video and voice, and extracts the video data and voice data during the interaction process. The video data includes the video screen, video format, screen size and other information when the user is conducting psychological consultation, which can be determined according to actual conditions, and the embodiment of the present invention does not make specific restrictions on this. The voice data includes the audio, audio format and other information when the user is conducting psychological consultation, which can be determined according to actual conditions, and the embodiment of the present invention does not make specific restrictions on this.

[0056] Since the video and audio formats generated by different devices may be different, in order to ensure that the intelligent psychological robot consultation system can process all video and audio data, the video and audio formats are converted into standard video and audio formats during preprocessing.

[0057] As an implementation manner, the steps of preprocessing the video data and the voice data respectively include:

[0058] Performing format conversion and denoising operations on the video data in sequence to obtain preprocessed video data;

[0059] Perform noise reduction and echo cancellation operations on the voice data to obtain pre-processed voice data.

[0060] In an embodiment of the present invention, format conversion and denoising operations are performed on video data in sequence. The format conversion refers to whether the format of the video data matches the standard format. If it does not match, the format of the video data is converted to the standard format, and denoising is performed on the video data to obtain pre-processed video data.

[0061] Then, the voice data is format converted and denoised. The format conversion refers to whether the format of the voice data matches the standard format. If it does not match, the format of the voice data is converted to the standard format, and the voice data is denoised, and the denoised audio is echo-cancelled to obtain the pre-processed voice data. The echo cancellation operation on the audio can be performed by using a filter.

[0062] S120, performing emotion recognition on the frame image corresponding to each time stamp of the preprocessed video data to obtain an image emotion recognition result corresponding to each time stamp;

[0063] When performing image recognition on video data, emotion recognition cannot be performed directly based on the video. Therefore, it is necessary to first extract frame images from the video data, set timestamps at a certain frequency, and then extract the frame images corresponding to each timestamp in the video data. Perform emotion recognition on each frame image separately to obtain the image emotion recognition results corresponding to each timestamp.

[0064] Among them, the emotion recognition result of the frame image can adopt the image feature extraction method or the deep learning method, which can be determined according to the actual situation and is not specifically limited in the embodiment of the present invention.

[0065] As an implementation manner, the steps of performing emotion recognition on the frame image corresponding to each timestamp of the preprocessed video data to obtain the image emotion recognition result corresponding to each timestamp include:

[0066] At least 3 timestamps are set per second, and frame images corresponding to each timestamp of the preprocessed video data are extracted;

[0067] The frame image corresponding to each timestamp is input into the visual language model to obtain the image emotion recognition result corresponding to each timestamp.

[0068] In the embodiment of the present invention, when collecting frame images of each timestamp in the video, at least 3 timestamps are set per second, so that the user's micro-expressions during the consultation process can be captured, and the user's expression changes can be accurately captured, thereby making the subsequent emotion recognition results more accurate.

[0069] In the embodiment of the present invention, a frame image corresponding to each timestamp in the video is collected, and the frame image corresponding to each timestamp is input into a visual language model to obtain an image emotion recognition result corresponding to each timestamp.

[0070] Among them, the Vision-Language Model (VLM) is a multimodal model that can learn from images and texts at the same time and output texts. The core of the Vision-Language Model is to fuse image information and text information. For images, an image encoder is used to convert them into feature vectors to extract key features of the image, such as color, shape, texture, etc.; for text, a text encoder is used to convert it into a text feature vector for subsequent fusion with image features.

[0071] After obtaining the feature vectors of the image and text, they need to be aligned and interacted by embedding the projection sub-model, etc. The purpose is to enable the visual language model to understand the relationship between the image and the text, for example, the objects in the image correspond to the objects described in the text. After feature fusion and interaction, the visual language model uses the text decoder to generate text output based on the fused features. This output can be a description of the image, an answer to image-related questions, an understanding and explanation of the image content, etc. In the embodiment of the present invention, the output of the visual language model is the image emotion recognition result.

[0072] It should be noted that the visual language model needs to be trained with samples and labels before use. In addition, the labels include fear, frustration, anxiety, anger, happiness, surprise, excitement, disgust and stress, and the image emotion recognition result corresponding to each timestamp is any one of these nine emotions.

[0073] S130, inputting the audio, the first task prompt word and the second task prompt word corresponding to each timestamp of the preprocessed audio data into a speech transcription emotion recognition integrated model to obtain a speech emotion recognition result corresponding to each timestamp, wherein the speech transcription emotion recognition integrated model is obtained after training with samples, the first task prompt word is the transcribed audio text, and the second task prompt word is the recognized audio emotion;

[0074] In the embodiment of the present invention, an audio segment corresponding to each timestamp in the preprocessed audio data is extracted. The audio segment may be a time segment with the timestamp as the center point. The specific length may be determined according to actual conditions, and the embodiment of the present invention does not specifically limit this.

[0075] After extracting the audio clip, the first task prompt word and the second task prompt word are generated. The first task prompt word is to transcribe the audio text, and the second task prompt word is to recognize the audio situation. When the first task prompt word and the second task prompt word are input into the speech transcription emotion recognition integrated model, the target task of the speech transcription emotion recognition integrated model is to convert the audio into text and generate the speech emotion recognition result corresponding to the audio.

[0076] It should be noted that the speech transcription and emotion recognition integrated model is a neural network model. Before using the speech transcription and emotion recognition integrated model, it is necessary to train it with samples, which may be audio samples, first task prompt words and second task prompt words.

[0077] As an implementation mode, the speech transcription emotion recognition integrated model includes an encoder, a text embedder and a decoder of an enhanced language model, and the audio corresponding to each timestamp of the preprocessed audio data, the first task prompt word and the second task prompt word are input into the speech transcription emotion recognition integrated model to obtain the speech emotion recognition result corresponding to each timestamp, and the steps include:

[0078] Input the audio corresponding to each timestamp into the encoder to obtain an audio vector;

[0079] Inputting the first task prompt word and the second task prompt word into the text embedder respectively to obtain a first task vector and a second task vector;

[0080] Splicing the audio vector and the first task vector to obtain a first splicing vector, and splicing the first task vector and the second task vector to obtain a second splicing vector;

[0081] The first splicing vector and the second splicing vector are input into the decoder to obtain the transcription text corresponding to each timestamp and the speech emotion recognition result corresponding to each timestamp.

[0082] Figure 2 A schematic diagram of the structure of a speech transcription and emotion recognition integrated model provided by an embodiment of the present invention, such as Figure 2 As shown in Figure 2, the speech transcription emotion recognition integrated model includes an encoder, a text embedder, and a decoder. The connection relationship can be referred to in Figure 2 As shown, the audio corresponding to each timestamp is input into the encoder, and the audio is encoded to obtain the audio component.

[0083] The encoder can encode both audio and text. The encoder can be a Transformer encoder, a CNN-RNN hybrid encoder, a VGGish encoder, etc. The specific encoder can be determined according to actual conditions, and the embodiment of the present invention does not make specific limitations on this.

[0084] For example, the Transformer encoder was originally used in natural language processing, and its core is the self-attention mechanism. For text encoding, it can capture the long-distance dependencies and semantic information between words in the text and convert the text into a low-dimensional vector representation. For audio encoding, the audio signal is usually first converted into a feature representation suitable for Transformer processing, such as a spectrogram, and then the Transformer encoder processes these features to learn the characteristic patterns of the audio.

[0085] It performs well in multimodal tasks, has strong feature extraction and modeling capabilities, and can adaptively focus on important parts of the input data, whether it is semantic key information in text or specific frequency features in audio, etc. It can also process input sequences of different lengths and has good adaptability to modalities with large length differences such as audio and text.

[0086] Input the first task prompt word and the second task prompt word into the text embedder respectively to obtain the first task vector and the second task vector. Among them, text embedding is a technical tool that converts text into numerical vectors, which plays an important role in the field of natural language processing. The basis of text embedding is to represent words or characters in text as numerical vectors. The idea of ​​distributed representation is to disperse the semantic information of words in various dimensions of the vector so that words with similar semantics are close in the vector space. For example, the words "cat" and "dog" may be close in the vector space because they are both animals and have some similar characteristics.

[0087] Then, the audio vector and the first task vector are concatenated to obtain a first concatenated vector, and the first task vector and the second task vector are concatenated to obtain a second concatenated vector. Vector concatenation is a commonly used operation in data processing and machine learning. It can combine multiple vectors into a longer vector. Vector concatenation refers to merging two or more vectors along a specific dimension to obtain a new vector whose length is equal to the sum of the lengths of the original vectors. When concatenating vectors, the concatenation dimension needs to be specified.

[0088] The first concatenated vector and the second concatenated vector are input into the decoder to obtain the transcribed text corresponding to each timestamp and the speech emotion recognition result corresponding to each timestamp. Among them, the decoder is the decoder of the enhanced language model. The decoder of the enhanced language model plays a key role in the entire language model, responsible for converting the knowledge and potential representation learned by the model into natural language output. The main task of the decoder is to generate natural language text word by word or word by word based on the input information and the internal state of the model. This includes generating text content of various lengths such as sentences, paragraphs, and articles.

[0089] In some enhanced language models, the decoder can process multimodal information, such as images, audio, etc. By fusing multimodal information with text information, the decoder can generate richer and more accurate text output.

[0090] It should be noted that the speech emotion recognition result includes fear, frustration, anxiety, anger, happiness, surprise, excitement, disgust, and stress, and the speech emotion recognition result output by the decoder is one of the above nine emotions.

[0091] In an embodiment of the present invention, the audio features extracted by the multiplexing encoder are spliced ​​with the task prompt features of text transcription and emotion recognition in the vector space, and then passed into the decoder to obtain the transcribed text and emotions. In this process, the audio encoder only needs to encode once, thereby greatly saving the computing and storage costs and improving the real-time performance of the analysis.

[0092] S140, obtaining a final emotion recognition result corresponding to each timestamp according to the image emotion recognition result corresponding to each timestamp and the speech emotion recognition result corresponding to each timestamp;

[0093] After obtaining the image emotion recognition result corresponding to each timestamp and the speech emotion recognition result corresponding to each timestamp, the two results are fused to obtain the final emotion recognition result corresponding to each timestamp.

[0094] As an implementation manner, the steps of obtaining a final emotion recognition result corresponding to each timestamp according to the image emotion recognition result corresponding to each timestamp and the speech emotion recognition result corresponding to each timestamp include:

[0095] For each timestamp, if the image emotion recognition result and the voice emotion recognition result are consistent, the image emotion recognition result / the voice emotion recognition result is used as the final emotion recognition result;

[0096] If the image emotion recognition result is inconsistent with the speech emotion recognition result, the final emotion recognition results corresponding to multiple timestamps before the current timestamp are obtained, and the emotion with the highest frequency is taken as the final emotion recognition result.

[0097] In an embodiment of the present invention, when fusing the image emotion recognition results and the speech emotion recognition results, for each timestamp, it is first determined whether the image emotion recognition result and the speech emotion recognition result corresponding to the timestamp are consistent. If they are consistent, the image emotion recognition result or the speech emotion recognition result is used as the final emotion recognition result; if the two are inconsistent, the previously identified emotion needs to be used at this time. Considering that emotions usually do not change suddenly, the final emotion recognition results corresponding to the first few timestamps of the timestamp are counted, and among the final emotion recognition results corresponding to the first few timestamps, the image emotion recognition results corresponding to the timestamp, and the speech emotion recognition results corresponding to the timestamp, the emotion with the highest frequency is counted, and the emotion with the highest frequency is used as the final emotion recognition result corresponding to the timestamp.

[0098] In the embodiment of the present invention, when the image emotion recognition result and the speech emotion recognition result are inconsistent, the image emotion recognition results appearing in the previous frames are combined, and the emotion with the highest frequency of occurrence is used as the final emotion recognition result, thereby further improving the accuracy of the emotion recognition result.

[0099] S150, generating an emotion state recognition report according to the final emotion recognition result corresponding to each timestamp, so as to perform psychological intervention on the user according to the emotion state recognition report.

[0100] The final emotion recognition result corresponding to each timestamp in the embodiment of the present invention generates an emotion state recognition report, which is a document that analyzes and summarizes the individual's emotion state, which can help people better understand their own emotion changes and the impact of emotions on life and work. The emotion state recognition report includes the frequency, intensity and cause of positive and negative emotions. By understanding the emotion state, it is hoped that some suggestions and strategies can be provided to help individuals better manage emotions and improve the quality of life and work efficiency.

[0101] When the emotional state recognition report shows that the user's emotions are relatively negative, psychological intervention is carried out on the user to help the user reduce the pressure caused by life, work, study, etc., so that they can face various challenges with a more relaxed attitude. For example, for users facing excessive work pressure, psychological intervention can help them learn effective coping strategies, such as time management, relaxation techniques, etc., to reduce stress levels; in addition, emotional states can be improved: negative emotions of users, such as anxiety, depression, anger, etc., are adjusted to promote the generation of positive emotions. For example, for users in a depressed state, psychological intervention can provide emotional support, help them identify and change negative thinking patterns, and improve emotional stability and positivity.

[0102] As an implementation mode, the steps of generating an emotion state recognition report according to the final emotion recognition result corresponding to each timestamp include:

[0103] Get every round of dialogue during human-computer interaction;

[0104] According to the occurrence time of each round of conversation and the final emotion recognition result corresponding to each timestamp, the real-time emotion corresponding to each round of conversation is obtained;

[0105] Integrate the conversation content and the real-time emotion of each round of conversation into the third task prompt words corresponding to each round of conversation;

[0106] Input the third prompt word corresponding to each round of dialogue into the large language model to obtain the emotion category corresponding to each round of dialogue;

[0107] The proportion of each emotion category is counted to generate the emotion state recognition report.

[0108] In the process of generating an emotion state recognition report in an embodiment of the present invention, each round of dialogue in the human-computer interaction process is first obtained, wherein a question and an answer can be regarded as a round of dialogue; then, according to the occurrence time of each round of dialogue, a search is performed in each timestamp to find a matching timestamp, and the final emotion recognition result corresponding to the matched timestamp is used as the real-time emotion corresponding to the round of dialogue; and the corresponding dialogue content and real-time emotion of each round of dialogue are used as the third task prompt word.

[0109] For example, the third task prompt is: Please judge the user's emotion based on the current conversation content and the user's emotion: rich emotion / weak emotion / lack of emotion / inappropriate emotion. Among them, "Please judge the user's emotion based on the current conversation content and the user's emotion" is the current conversation content, and "rich emotion / weak emotion / lack of emotion / inappropriate emotion" is the real-time emotion.

[0110] Finally, the third prompt word is input into the large language model to obtain the emotion category corresponding to each round of dialogue. In addition, the proportion of each emotion in all timestamps is counted, and an emotion state recognition report is output based on the proportion of each emotion.

[0111] Among them, a large language model is a language model built by a deep neural network with more than tens of billions of parameters. It is usually trained with a large amount of unlabeled text using self-supervised learning methods, and can predict and generate text and other content. Most large language models are based on the Transformer architecture. The self-attention mechanism in this architecture allows the model to adjust the representation of each position according to other positions in the sequence, thereby improving the model's ability to understand the context. Pre-training uses a large amount of unlabeled data to allow the model to learn the statistical patterns and semantic information of the language. The source of pre-training data is very wide, such as text, news, blogs, forums, etc. on the Internet. After completing the pre-training, the model is further trained using a small amount of high-quality data sets through instruction fine-tuning to explore and enhance the capabilities of the model itself, so that it performs better on specific tasks.

[0112] The present invention proposes a psychological intervention method based on multimodal emotion recognition, which first pre-processes the video data and voice data generated in the human-computer interaction process; then performs emotion recognition on the frame image corresponding to each timestamp in the pre-processed video to obtain the image emotion recognition result corresponding to each timestamp; then inputs the audio, the first task prompt word and the second task prompt word corresponding to each timestamp into the speech transcription emotion recognition integrated model to obtain the speech emotion recognition result corresponding to each timestamp; finally, the image emotion recognition result and the speech emotion recognition result are fused to obtain the final emotion recognition result, and an emotion state recognition report is generated to perform psychological intervention on the user.

[0113] Since emotion recognition of a single modality may have deviations, the present invention performs emotion recognition based on video data and voice data respectively, and integrates the two emotion recognition results to obtain the final emotion recognition result, making the emotion recognition result more accurate; in addition, when traditional speech is transcribed into text and emotion recognition is performed, computational and storage redundancy is generated. The integrated speech transcription and emotion recognition model proposed in the present invention reuses the audio features extracted by the encoder. In this process, the audio encoder only needs to encode once, thereby greatly saving computational and storage costs and improving the real-time performance of the analysis.

[0114] Figure 3 A schematic diagram of the structure of a psychological intervention system based on multimodal emotion recognition provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the system includes:

[0115] The preprocessing module 310 is used to obtain the video data and voice data generated during the human-computer interaction process, and preprocess the video data and the voice data respectively;

[0116] The image recognition module 320 is used to perform emotion recognition on the frame image corresponding to each time stamp of the pre-processed video data to obtain the image emotion recognition result corresponding to each time stamp;

[0117] The speech recognition module 330 is used to input the audio, the first task prompt word and the second task prompt word corresponding to each timestamp of the preprocessed audio data into the speech transcription emotion recognition integrated model to obtain the speech emotion recognition result corresponding to each timestamp, wherein the speech transcription emotion recognition integrated model is obtained after training with samples, the first task prompt word is the transcribed audio text, and the second task prompt word is the recognized audio emotion;

[0118] The fusion module 340 is used to obtain a final emotion recognition result corresponding to each timestamp according to the image emotion recognition result corresponding to each timestamp and the speech emotion recognition result corresponding to each timestamp;

[0119] The intervention module 350 is used to generate an emotional state recognition report according to the final emotion recognition result corresponding to each timestamp, so as to perform psychological intervention on the user according to the emotional state recognition report.

[0120] This embodiment is a system embodiment corresponding to the above method embodiment, and its implementation process is the same as that of the above method embodiment. For details, please refer to the above method embodiment, and this system embodiment does not make specific limitations on this.

[0121] Each module in the above-mentioned psychological intervention system based on multimodal emotion recognition can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0122] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention, the computer device may be a server, and its internal structure diagram may be as follows Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a computer storage medium and an internal memory. The computer storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the computer storage medium. The database of the computer device is used to store data generated or obtained during the execution of a psychological intervention method based on multimodal emotion recognition, such as video data and voice data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a psychological intervention method based on multimodal emotion recognition is implemented.

[0123] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of a psychological intervention method based on multimodal emotion recognition in the above embodiment are implemented. Alternatively, when the processor executes the computer program, the functions of each module / unit in the embodiment of a psychological intervention system based on multimodal emotion recognition are implemented.

[0124] In one embodiment, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of a psychological intervention method based on multimodal emotion recognition in the above embodiment are implemented. Alternatively, when the computer program is executed by a processor, the functions of each module / unit in the embodiment of the psychological intervention system based on multimodal emotion recognition are implemented.

[0125] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0126] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0127] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A psychological intervention method based on multimodal emotion recognition, characterized in that: include: Acquire video data and voice data generated during the human-computer interaction process, and pre-process the video data and voice data respectively; Performing emotion recognition on the frame image corresponding to each time stamp of the preprocessed video data to obtain the image emotion recognition result corresponding to each time stamp; Input the audio, the first task prompt word and the second task prompt word corresponding to each timestamp of the preprocessed audio data into the speech transcription emotion recognition integrated model to obtain the speech emotion recognition result corresponding to each timestamp, wherein the speech transcription emotion recognition integrated model is obtained after training with samples, the first task prompt word is the transcribed audio text, and the second task prompt word is the recognized audio emotion; According to the image emotion recognition result corresponding to each timestamp and the speech emotion recognition result corresponding to each timestamp, a final emotion recognition result corresponding to each timestamp is obtained; Generate an emotion state recognition report according to the final emotion recognition result corresponding to each timestamp, so as to conduct psychological intervention on the user according to the emotion state recognition report; The speech transcription emotion recognition integrated model includes an encoder, a text embedder and a decoder of an enhanced language model. The audio corresponding to each timestamp of the preprocessed audio data, the first task prompt word and the second task prompt word are input into the speech transcription emotion recognition integrated model to obtain the speech emotion recognition result corresponding to each timestamp. The steps include: Input the audio corresponding to each timestamp into the encoder to obtain an audio vector; Inputting the first task prompt word and the second task prompt word into the text embedder respectively to obtain a first task vector and a second task vector; Splicing the audio vector and the first task vector to obtain a first splicing vector, and splicing the first task vector and the second task vector to obtain a second splicing vector; Inputting the first splicing vector and the second splicing vector into the decoder to obtain a transcription text corresponding to each timestamp and a speech emotion recognition result corresponding to each timestamp; The steps of obtaining a final emotion recognition result corresponding to each timestamp according to the image emotion recognition result corresponding to each timestamp and the speech emotion recognition result corresponding to each timestamp include: For each timestamp, if the image emotion recognition result and the voice emotion recognition result are consistent, the image emotion recognition result / the voice emotion recognition result is used as the final emotion recognition result; If the image emotion recognition result is inconsistent with the speech emotion recognition result, obtaining final emotion recognition results corresponding to multiple timestamps before the current timestamp, and taking the emotion with the highest frequency as the final emotion recognition result; The steps of generating an emotion state recognition report according to the final emotion recognition result corresponding to each timestamp include: Get every round of dialogue during human-computer interaction; According to the occurrence time of each round of conversation and the final emotion recognition result corresponding to each timestamp, the real-time emotion corresponding to each round of conversation is obtained; Integrate the conversation content and the real-time emotion of each round of conversation into the third task prompt words corresponding to each round of conversation; Input the third prompt word corresponding to each round of dialogue into the large language model to obtain the emotion category corresponding to each round of dialogue; The proportion of each emotion category is counted to generate the emotion state recognition report.

2. The psychological intervention method based on multimodal emotion recognition according to claim 1, characterized in that: The image emotion recognition results and the voice emotion recognition results include fear, frustration, anxiety, anger, happiness, surprise, excitement, disgust, and stress; The emotion categories include rich emotion, weak emotion, lack of emotion, and inappropriate emotion.

3. The psychological intervention method based on multimodal emotion recognition according to claim 1, characterized in that: The steps of performing emotion recognition on the frame image corresponding to each time stamp of the preprocessed video data to obtain the image emotion recognition result corresponding to each time stamp include: At least 3 timestamps are set per second, and frame images corresponding to each timestamp of the preprocessed video data are extracted; The frame image corresponding to each timestamp is input into the visual language model to obtain the image emotion recognition result corresponding to each timestamp.

4. The psychological intervention method based on multimodal emotion recognition according to any one of claims 1 to 3, characterized in that: The steps of preprocessing the video data and the voice data respectively include: Performing format conversion and denoising operations on the video data in sequence to obtain preprocessed video data; Perform noise reduction and echo cancellation operations on the voice data to obtain pre-processed voice data.

5. A psychological intervention system based on multimodal emotion recognition, characterized in that: include: A preprocessing module, used to obtain video data and voice data generated during the human-computer interaction process, and preprocess the video data and voice data respectively; An image recognition module is used to perform emotion recognition on the frame image corresponding to each time stamp of the preprocessed video data to obtain the image emotion recognition result corresponding to each time stamp; A speech recognition module is used to input the audio, the first task prompt word and the second task prompt word corresponding to each timestamp of the preprocessed audio data into a speech transcription emotion recognition integrated model to obtain a speech emotion recognition result corresponding to each timestamp, wherein the speech transcription emotion recognition integrated model is obtained after training with samples, the first task prompt word is the transcribed audio text, and the second task prompt word is the recognized audio emotion; A fusion module, used to obtain a final emotion recognition result corresponding to each timestamp according to the image emotion recognition result corresponding to each timestamp and the speech emotion recognition result corresponding to each timestamp; An intervention module, used to generate an emotional state recognition report according to the final emotion recognition result corresponding to each timestamp, so as to perform psychological intervention on the user according to the emotional state recognition report; The speech transcription emotion recognition integrated model includes an encoder, a text embedder and a decoder of an enhanced language model. The audio corresponding to each timestamp of the preprocessed audio data, the first task prompt word and the second task prompt word are input into the speech transcription emotion recognition integrated model to obtain the speech emotion recognition result corresponding to each timestamp. The steps include: Input the audio corresponding to each timestamp into the encoder to obtain an audio vector; Inputting the first task prompt word and the second task prompt word into the text embedder respectively to obtain a first task vector and a second task vector; Splicing the audio vector and the first task vector to obtain a first splicing vector, and splicing the first task vector and the second task vector to obtain a second splicing vector; Inputting the first splicing vector and the second splicing vector into the decoder to obtain a transcription text corresponding to each timestamp and a speech emotion recognition result corresponding to each timestamp; The steps of obtaining a final emotion recognition result corresponding to each timestamp according to the image emotion recognition result corresponding to each timestamp and the speech emotion recognition result corresponding to each timestamp include: For each timestamp, if the image emotion recognition result and the voice emotion recognition result are consistent, the image emotion recognition result / the voice emotion recognition result is used as the final emotion recognition result; If the image emotion recognition result is inconsistent with the speech emotion recognition result, obtaining final emotion recognition results corresponding to multiple timestamps before the current timestamp, and taking the emotion with the highest frequency as the final emotion recognition result; The steps of generating an emotion state recognition report according to the final emotion recognition result corresponding to each timestamp include: Get every round of dialogue during human-computer interaction; According to the occurrence time of each round of conversation and the final emotion recognition result corresponding to each timestamp, the real-time emotion corresponding to each round of conversation is obtained; Integrate the conversation content and the real-time emotion of each round of conversation into the third task prompt words corresponding to each round of conversation; Input the third prompt word corresponding to each round of dialogue into the large language model to obtain the emotion category corresponding to each round of dialogue; The proportion of each emotion category is counted to generate the emotion state recognition report.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the psychological intervention method based on multimodal emotion recognition as described in any one of claims 1 to 4 are implemented.

7. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the psychological intervention method based on multimodal emotion recognition as claimed in any one of claims 1 to 4 are implemented.

Images