Emotion recognition method based on multi-modal large model feature fusion
Through the fusion of multimodal large model features, combining cameras and recording devices to collect data, extract and fuse word vectors, the limitations of traditional single-modal emotion recognition methods are solved, and more efficient emotion recognition performance and lower data requirements are achieved. .
Patent Information
- Application Number
- CN202411354254.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2025-01-14
AI Technical Summary
Traditional emotion recognition methods are usually based on a single modal, rely on features of artificial design, have poor results and are susceptible to feature design, and require a large amount of labeled data.
Using the multimodal big model feature fusion method, facial expressions and voice data are collected through cameras and recording devices, converted into word vectors, and features are extracted using images, voice and text models to fusion and enhance them, and finally emotionally performed through classifiers Classification.
It improves the performance of emotion recognition, reduces the dependence on domain expertise, reduces the cost of data collection and labeling, and achieves stronger versatility and generalization.
Smart Images
Figure CN119312043A_ABST
Abstract
Description
Technical Field
[0001] The present invention mainly relates to the technical field of emotion recognition, and specifically to an emotion recognition method based on multimodal large model feature fusion. Background Art
[0002] Traditional emotion recognition methods often judge from one aspect of images, speech or text, that is, unimodal methods, and also need to design features based on human experience for different modalities. This is generally not very effective, and the final performance is easily affected by the quality of the manually designed features. In the past two years, the large model technology in the field of artificial intelligence has made great progress. By pre-training large models on a large amount of unsupervised corpus, very low-level and common features of different modalities can be captured. This feature extracted by the large model has strong generalization and can be applied to different downstream tasks, and no professional knowledge in the field is required to design features. Therefore, this patent proposes a method of integrating the features extracted by the three different modal large models, fusing and enhancing them, and then classifying emotions, which can achieve better performance than unimodal and manually designed features. At the same time, with the help of the powerful expression ability of the large model, we do not need to collect a large amount of labeled data for specific fields as before. Now only a small amount of data needs to be collected, and the model G used for classification is simply fine-tuned. Summary of the invention
[0003] The present invention mainly provides an emotion recognition method based on multimodal large model feature fusion to solve the technical problems raised in the above background technology.
[0004] The technical solution adopted by the present invention to solve the above technical problems is:
[0005] A method for emotion recognition based on multimodal large model feature fusion, comprising the following steps:
[0006] Step 1: Use a camera to record the changes in the facial expressions of the collectors, and use a recording device to record all the corpus information, and use the open source speech recognition engine to obtain the corresponding text from the audio corpus;
[0007] Step 2: Obtain three sets of word vectors using the video, audio data and text information obtained in step 1;
[0008] Step 3: The three groups of word vectors are fused in an enhanced manner, and then connected to the classifier to train the sentiment classification model. The trained sentiment classification model is stored locally, and the model is recorded as G.
[0009] Step 4: In the actual application scenario, a wide-angle camera is arranged to capture all persons to be identified; at the same time, a high-quality microphone array pickup device is arranged at the top of the scene;
[0010] Step 5: Get the key frame images, audio, and transcribed text, and send them to the same image model, speech model, and text model through the network. Wait a few seconds until the three cloud models are processed and return the embedded word vectors extracted by the three local models.
[0011] In step six, the embedded word vectors corresponding to the three large models are comprehensively enhanced in the same way as in step 3, and then sent to the local classification model G. The model ultimately determines the student’s emotional state at this time and returns the state to the monitoring personnel.
[0012] Furthermore, in step 1, before recording the changes in the facial expressions of the collectors with a camera, the training data is recorded. Since the pre-trained large model has strong expressive power, the data for downstream tasks does not need to record a large amount of corpus like the traditional method. For specific tasks, such as emotion recognition of college students, 20 young men and women aged 18-23 can be arranged, each of whom speaks 100 sentences for different emotion categories to ensure that the expression and voice tone are consistent with the emotions they really want to express. The text content is only auxiliary, allowing the same text to express different emotions in different ways of speaking.
[0013] Furthermore, in step 2, the key frames of the video of facial expression changes are extracted and converted into image information, and the embedded word vectors vector1 corresponding to the image information are obtained through an open source image model. The audio data is passed through an open source speech model to obtain the corresponding word vector vector2. The text information is passed through an open source text model to obtain a word vector vector3, and the open source speech model includes wav2vec2 or HuBERT.
[0014] Furthermore, in step three, the three groups of word vectors are multiplied by a projection matrix respectively, the dimensions are adjusted to be consistent, and then added; or several layers of Transformer Layer are added to capture global features, so as to fuse the three groups of word vectors in an enhanced manner.
[0015] Furthermore, all systems in step 4 are run on an embedded device local to the scene, which has a certain computing power sufficient to support a lightweight local speech recognition engine, and the device can be connected to the Internet and can access the Internet API interface through get / post requests.
[0016] Furthermore, in the step four, when the face of the person to be identified changes, the camera automatically focuses on the face of the person to be identified, adjusts the focal length of the camera to capture the changes in the facial expressions of the person to be identified as clearly as possible, records the video of the facial expressions of the person to be identified, and uses the microphone to record the voice of the person to be identified. The microphone has the functions of sound source positioning and beam forming. When the position of the person to be identified is captured, the voice from that direction will be automatically enhanced and the voice from other directions will be weakened, so as to better record the voice of the person to be identified.
[0017] Furthermore, the key frames are extracted from the video of the facial expression changes of the person to be identified, and the extraction logic is consistent with that during training. At the same time, the recorded voice of the person to be identified is quickly transcribed by the local voice recognition engine.
[0018] Furthermore, the wide-angle camera in step 4 is connected to motors (11) on both sides, the output shaft of the motor (11) is connected to an arc-shaped bar (12), and the inner side of the arc-shaped bar (12) is connected to a brush (13).
[0019] Compared with the prior art, the present invention has the following beneficial effects:
[0020] 1. It integrates the features extracted from three different modal large models, avoiding the one-sidedness of single-dimensional features, and the features of different modalities can complement and enhance each other.
[0021] 2. The output of the pre-trained large model replaces the traditional method of setting features based on manual experience. There is no need to introduce additional domain expertise, which lowers the threshold of the task. At the same time, the features extracted by the large model have stronger versatility.
[0022] 3. Only a small amount of labeled data is needed to train the final lightweight classification network. There is no need to collect a large amount of labeled data for specific tasks as in traditional methods, which reduces time and labor costs.
[0023] The present invention will be explained in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 It is a structural schematic diagram of the wide-angle camera of the present invention;
[0025] Figure 2 It is a schematic structural diagram of the arc strip of the present invention.
[0027] In the figure: 11, motor; 12, arc strip; 13, brush. DETAILED DESCRIPTION
[0028] To facilitate understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings, but the present invention can be implemented in different forms and is not limited to the embodiments described in the text. On the contrary, these embodiments are provided to make the content disclosed in the present invention more thorough and comprehensive.
[0029] It should be noted that when an element is referred to as being "fixed on" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used in this article are for illustrative purposes only.
[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which the present invention belongs. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.
[0031] For examples, please refer to the attached Figure 1-2 , the emotion recognition method based on multimodal large model feature fusion is characterized in that the specific process includes the following steps:
[0032] Step 1: First, record a batch of training data. Since the pre-trained large model has powerful expressive power, the data for downstream tasks does not need to record a large amount of corpus like traditional methods. For specific tasks, such as emotion recognition of college students, 20 young men and women aged 18-23 can be arranged. Each person says 100 sentences for different emotion categories. It is necessary to ensure that the expression and voice tone are consistent with the emotion they really want to express. The text content is only auxiliary, allowing the same text to express different emotions in different ways of speaking. Use a camera to record the changes in the speaker's facial expressions, and use a recording device to record all the corpus information. Use an open source speech recognition engine to obtain the corresponding text for the audio corpus, or let the speaker directly say the given text, thereby omitting the speech recognition step. Organize and store the data (facial expression video, audio, text, emotion category).
[0033] Step 2: Extract key frames from the video of facial expression changes and convert them into image information. Use the open-source image model to obtain the embedded word vector vector1 corresponding to these image information. Use the open-source speech model (wav2vec2 or HuBERT) to obtain the corresponding word vector vector2 for the audio data. Use the open-source text model to obtain the word vector vector3 for the text information.
[0034] Step 3: Fuse the three word vectors [vector1, vector2, vector3] in an enhanced way, such as multiplying each by a projection matrix, adjusting the dimensions to be consistent, and then adding them together; or connect a few layers of Transformer Layer to capture global features. Then connect the classifier to train the sentiment classification model. The classifier here can be a traditional linear classifier, Logistic regression, or a few layers of fully connected neural network. Store the trained sentiment classification model locally, and the model is recorded as G.
[0035] Step 4: In actual application scenarios, such as smart classrooms on university campuses, wide-angle cameras that can capture all students are deployed; at the same time, high-quality microphone array pickup devices are deployed on the top of the classroom. All systems run on local embedded devices in the classroom, such as embedded Linux systems. The device has a certain computing power that is sufficient to support a lightweight local speech recognition engine, and the device can be connected to the Internet and can access the Internet API interface through get / post requests.
[0036] Step 5: When the teacher calls a student to answer a question during class, the camera automatically focuses on the student's face and adjusts the focal length of the camera to capture the changes in the student's facial expression as clearly as possible. Record the video of the student's facial expression. At the same time, use the microphone to record the student's voice. The microphone has the functions of sound source localization and beamforming. When the student's position is captured, the voice from that direction will be automatically enhanced and the voice from other directions will be weakened, so as to better record the student's voice.
[0037] Step 6: Extract key frames from the video of the student's facial expression changes. The extraction logic is basically the same as that during training. At the same time, the recorded student voice is quickly transcribed by the local speech recognition engine. The key frame images, audio, and transcribed text are sent to the same image model, speech model, and text model as during training through the network. Wait a few seconds for the three cloud models to complete processing and return the embedded word vectors extracted by the three local models.
[0038] Step 7: The embedded word vectors corresponding to the three large models are comprehensively enhanced in the same way as in step 3, and then sent to the local classification model G. The model will ultimately determine the student’s emotional state at this time and return the state to the teacher, for example, by displaying it in real time on the teacher’s computer screen at the podium, so that the teacher can further optimize the content and rhythm of the class according to the student’s emotional state.
[0039] The above is an exemplary description of the present invention in combination with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned method. As long as such non-substantial improvements are made by adopting the method concept and technical solution of the present invention, or the concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.
Claims
1. A method for emotion recognition based on multimodal large model feature fusion, characterized in that: The following steps are involved: Step 1: Use a camera to record the changes in the facial expressions of the collectors, and use a recording device to record all the corpus information, and use the open source speech recognition engine to obtain the corresponding text from the audio corpus; Step 2: Obtain three sets of word vectors using the video, audio data and text information obtained in step 1; Step 3: The three groups of word vectors are fused in an enhanced manner, and then connected to the classifier to train the sentiment classification model. The trained sentiment classification model is stored locally, and the model is recorded as G. Step 4: In the actual application scenario, a wide-angle camera is arranged to capture all persons to be identified; at the same time, a high-quality microphone array pickup device is arranged at the top of the scene; Step 5: Get the key frame images, audio, and transcribed text, and send them to the same image model, speech model, and text model through the network. Wait a few seconds until the three cloud models are processed and return the embedded word vectors extracted by the three local models. In step six, the embedded word vectors corresponding to the three large models are comprehensively enhanced in the same way as in step 3, and then sent to the local classification model G. The model ultimately determines the student’s emotional state at this time and returns the state to the monitoring personnel.
2. The emotion recognition method based on multimodal large model feature fusion according to claim 1 is characterized in that: In the step 1, before using a camera to record the changes in the facial expressions of the collector, training data is recorded to allow the same text to express different emotions in different ways of speaking.
3. The emotion recognition method based on multimodal large model feature fusion according to claim 1 is characterized in that: In the step 2, the key frames of the video of facial expression changes are extracted and converted into image information, and the embedded word vectors vector1 corresponding to the image information are obtained through the open source image model. The audio data is passed through the open source speech model to obtain the corresponding word vector vector2. The text information is passed through the open source text model to obtain the word vector vector3, and the open source speech model includes wav2vec2 or HuBERT.
4. The emotion recognition method based on multimodal large model feature fusion according to claim 1 is characterized in that: In the step three, the three groups of word vectors are multiplied by a projection matrix respectively, the dimensions are adjusted to be consistent, and then added; or several layers of Transformer Layer are added to capture global features, so as to fuse the three groups of word vectors in an enhanced manner.
5. The emotion recognition method based on multimodal large model feature fusion according to claim 1 is characterized in that: All systems in step 4 are run on an embedded device local to the scene. The device has a certain computing power, which is sufficient to support a lightweight local speech recognition engine. The device can be connected to the Internet and can access the Internet API interface through get / post requests.
6. The emotion recognition method based on multimodal large model feature fusion according to claim 1 is characterized in that: In the step four, when the face of the person to be identified changes, the camera automatically focuses on the face of the person to be identified, adjusts the focal length of the camera to capture the changes in the facial expressions of the person to be identified as clearly as possible, records a video of the facial expressions of the person to be identified, and uses a microphone to record the voice of the person to be identified. The microphone has the functions of sound source localization and beamforming. When the position of the person to be identified is captured, the voice from that direction will be automatically enhanced and the voice from other directions will be weakened, so as to better record the voice of the person to be identified.
7. The emotion recognition method based on multimodal large model feature fusion according to claim 1 is characterized in that: The key frames are extracted from the video of the facial expression changes of the person to be identified, and the extraction logic is consistent with that during training. At the same time, the recorded voice of the person to be identified is quickly transcribed into the local speech recognition engine.
8. The emotion recognition method based on multimodal large model feature fusion according to claim 1 is characterized in that: The wide-angle camera in step 4 is connected to motors (11) on both sides, the output shaft of the motor (11) is connected to an arc-shaped bar (12), and the inner side of the arc-shaped bar (12) is connected to a brush (13).