Emotion recognition method and related device, intelligent cabin and storage medium
The method integrates behavioral and environmental data to enhance emotion recognition accuracy and applicability by considering the target object's environment, addressing the limitations of existing methods.
Patent Information
- Application Number
- CN202410916403.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-12-03
AI Technical Summary
Existing emotion recognition methods struggle with accuracy and applicability, particularly when target objects express subtle emotions, limiting their scope of application.
An emotion recognition method that integrates behavioral action data with multimodal data from the target object's environment, using feature extraction and fusion techniques to predict emotions more accurately by considering environmental induction factors.
Improves the accuracy and applicability of emotion recognition by considering both the behavior of the target object's behavior and the environment, the method enhances the accuracy of the method by incorporating behavioral and environmental factors, allowing the target object's emotions, and the scene where the target object is located, thereby improving the accuracy and applicability of emotion recognition.
Smart Images

Figure CN119066604A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of emotion recognition, and in particular to an emotion recognition method and related devices, a smart cockpit and a storage medium. Background Art
[0002] With the widespread implementation of human-computer interaction technology, machine perception of emotions is of great significance for optimizing the human-computer interaction experience. For example, if machines can accurately understand and respond to human emotions, it will help greatly improve user experience and satisfaction.
[0003] At present, the existing methods for emotion recognition of target objects are usually achieved by analyzing the behavior of the target objects. However, when the target objects express subtle emotions, the existing methods will affect the accuracy of emotion recognition. In other words, the scope of application of the existing methods is relatively narrow. In view of this, how to improve the accuracy and applicability of emotion recognition has become an urgent problem to be solved. Summary of the invention
[0004] The main technical problem solved by the present application is to provide an emotion recognition method and related devices, a smart cockpit and a storage medium, which can improve the accuracy and applicability of emotion recognition.
[0005] In order to solve the above technical problems, the first aspect of the present application provides an emotion recognition method, including: obtaining behavioral action data of a target object, and obtaining multimodal data of a scene in which the target object is located; extracting behavioral action features based on the behavioral action data, and extracting multimodal features based on the multimodal data; obtaining target emotion features by fusing at least the behavioral action features and the multimodal features; and predicting the target emotion of the target object based on the target emotion features.
[0006] In order to solve the above-mentioned technical problems, the second aspect of the present application provides an emotion recognition device, including: a data acquisition module, a feature extraction module, a feature fusion module and an emotion prediction module, the data acquisition module is used to acquire the behavior and action data of the target object, and acquire the multimodal data of the scene in which the target object is located; the feature extraction module is used to extract behavior and action features based on the behavior and action data, and to extract multimodal features based on the multimodal data; the feature fusion module is used to fuse at least the behavior and action features and the multimodal features to obtain the target emotion features; the emotion prediction module is used to predict the target emotion of the target object based on the target emotion features.
[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, which at least includes a memory and a processor coupled to each other, the memory stores program instructions, and the processor is used to execute the program instructions to implement the emotion recognition method in the above first aspect.
[0008] In order to solve the above-mentioned technical problems, the fourth aspect of the present application provides an intelligent cockpit, which at least includes a collection device and a control device communicatively connected to the collection device, the collection device is used to collect behavioral action data of the target object and multimodal data of the environment in which the target object is located, and the control device is the electronic device in the above-mentioned third aspect.
[0009] In order to solve the above technical problems, the fifth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, and the program instructions are used to implement the emotion recognition method of the first aspect.
[0010] The above scheme obtains the behavior data of the target object and the multimodal data of the scene where the target object is located, and then extracts the behavior features based on the behavior data, and extracts the multimodal features based on the multimodal data, so as to obtain the target emotion features based on at least the behavior features and the multimodal features, and then predict the target emotion of the target object based on the target emotion features. Therefore, in the process of emotion recognition, not only the behavior data of the target object is taken into consideration, but also the scene where the target object is located is further taken into consideration, which can fully consider the induction factors of the target object's environment on the target object's emotions, and the scene where the target object is located is represented by multimodal data, which can express the induction factors as richly as possible from different modal angles, and then integrate the behavior features and multimodal features, which can fully model the emotional impact of the scene where the target object is located as an induction factor on the behavior of the target object. Therefore, the accuracy and applicability of emotion recognition can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 It is a flowchart of an embodiment of the emotion recognition method of the present application;
[0012] Figure 2 is a schematic diagram of an embodiment of a first self-attention mechanism;
[0013] Figure 3 is a schematic diagram of a framework of an embodiment of an emotion recognition model;
[0014] Figure 4 is a schematic diagram for constructing the first sample data;
[0015] Figure 5 It is a schematic diagram of the framework of an embodiment of the emotion recognition device of the present application;
[0016] Figure 6 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;
[0017] Figure 7 It is a schematic diagram of the framework of an embodiment of the smart cockpit of the present application;
[0018] Figure 8 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION
[0019] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.
[0020] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.
[0021] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.
[0022] See also Figure 1 , Figure 1 It is a flowchart of an embodiment of the emotion recognition method of the present application.
[0023] Specifically, the following steps may be included:
[0024] Step S11: Obtain the behavior data of the target object, and obtain multimodal data of the scene in which the target object is located.
[0025] In an implementation scenario, the behavioral action data may include but is not limited to at least one of the following: video data recorded on the target object, audio data collected from the target object. It should be noted that before collecting the behavioral action data of the target object, you can first request the target object to grant data collection permissions. Exemplarily, the scope, purpose, and processing of data collection and other related information can be stated in the form of a pop-up box, etc., and the target object can be prompted to read and understand, in order to request the target object to grant data collection permissions, so as to collect data after obtaining the target object's authorization. In addition, the video data recorded on the target object may include but is not limited to the target object's facial expressions, body movements, etc. The audio data collected from the target object may include but is not limited to the target object's conversations, etc., and the specific content of the behavioral action data is not limited here.
[0026] In one implementation scenario, multimodal data includes but is not limited to at least one of the following: image data captured in the scene where the target object is located, audio data of the target object's environment, and demand data input by the target object in the environment. It should be noted that before collecting multimodal data of the scene where the target object is located, you can first request the data collection permission granted by the target object. For details, please refer to the above-mentioned related descriptions, which will not be repeated here. Taking emotion recognition during driving as an example, the multimodal data of the scene where the target object is located may include but is not limited to: images captured in front, behind or in other directions of the vehicle, ambient sounds around the vehicle, demand instructions issued by the target object, etc. Other application scenarios can be deduced by analogy, and examples will not be given one by one here.
[0027] Step S12: extracting behavioral action features based on the behavioral action data, and extracting multimodal features based on the multimodal data.
[0028] In an implementation scenario, taking the example that the behavior action data includes video data recorded for the target object and audio data collected for the target object, after obtaining the behavior action data, video encoding can be performed based on the video data recorded for the target object in the behavior action data to obtain a video encoding representation, and audio encoding can be performed based on the audio data collected for the target object in the behavior action data to obtain an audio encoding representation. On this basis, the video encoding representation and the audio encoding representation can be fused to obtain the behavior action feature. The above method can improve the accuracy of the behavior action feature by performing video encoding and audio encoding separately, and fusing the video encoding representation and the audio encoding representation to obtain the behavior action feature.
[0029] In a specific implementation scenario, in order to improve the efficiency of video coding, a video encoder can be used for video coding. As a possible example, the video encoder can be a CLIP image encoder followed by a temporal encoder consisting of several (e.g., 3, 4, etc.) Transformer layers. Of course, the above example is only a possible example in the actual application process and is not limited here. For example, the video encoder can also be a network model such as ViViT for encoding videos, which will not be given one by one here.
[0030] In a specific implementation scenario, in order to improve the audio coding efficiency, an audio encoder may be used for audio coding. As a possible example, the audio encoder may specifically use BEATs. Of course, the above example is only a possible example and is not limited here. For example, the audio encoder may also be a network model such as LSTM for audio coding, which will not be given one by one here.
[0031] In a specific implementation scenario, after obtaining the video coding representation and the audio coding representation, the video coding representation and the audio coding representation can be spliced and other operations can be performed to achieve fusion to obtain the behavior action feature. For example, taking splicing to achieve fusion as an example, the behavior action feature can be expressed as:
[0032] f action =concat[F video ,F audio ]……(1)
[0033] In the above formula (1), f action Indicates the characteristics of the behavior, F video represents the video coding representation, F audio Represents audio encoding representation, and concat represents concatenation operation.
[0034] In one implementation scenario, after obtaining multimodal data, the initial feature representation of each modality can be extracted in the same semantic space based on each modal data in the multimodal data, and then the initial feature representation of each modality is processed at least based on the self-attention mechanism to obtain multimodal features. The above method can enhance the feature information related to each other between the modalities by processing the initial feature representation of each modality through self-attention, which helps to improve the accuracy of modeling emotional inducements.
[0035] In a specific implementation scenario, in order to improve the efficiency of multimodal feature extraction, a neural network model such as ImageBind can be used to process multimodal data to extract the feature expression of each modality in the same semantic space, which can be used as the initial feature representation of each modality.
[0036] In a specific implementation scenario, for the specific process of processing the initial attention mechanism of each modality based on the self-attention mechanism, please refer to the technical details of the attention mechanism, which will not be repeated here.
[0037] In a specific implementation scenario, after being processed by the self-attention mechanism, it can also continue to be processed by, for example, a feedforward neural network to extract deep features as multimodal features.
[0038] Step S13: fusing at least the behavior action feature and the multimodal feature to obtain the target emotion feature.
[0039] In an implementation scenario, after obtaining the behavioral action features and multimodal features, the behavioral action features and the multimodal features can be directly subjected to operations such as splicing, weighting, and averaging to achieve fusion, and obtain the target emotion features, so as to comprehensively express the target emotion features that characterize the emotion-related characteristic information of the target object in terms of both behavioral actions and emotional inducements.
[0040] In another implementation scenario, different from the aforementioned implementation, as another possible fusion method, the first query feature can be initialized first, and then based on the first query feature, feature information related to emotion inducements is extracted from the multimodal features to obtain the second query feature, thereby obtaining the target emotion feature based on the second query feature and the behavior feature, and the second query feature contains the feature information related to emotion inducements in the multimodal features. In the above method, feature information related to emotion inducements is first extracted from the multimodal features through the initialized first query feature, and then the target emotion feature is obtained by combining the behavior feature. The first query feature can filter out feature information in the multimodal features that is not related to emotion inducements as much as possible, which helps to improve the accuracy of the target emotion feature.
[0041] In a specific implementation scenario, the first query feature can be obtained by random initialization. In order to facilitate the subsequent extraction of feature information related to emotion inducements from multimodal features based on the first query feature, the first query feature can have the same feature dimension as the multimodal feature, such as both can be recorded as d dimensions. In addition, the specific number of the first query feature can be adjusted during the experiment using sample data, and is not limited here.
[0042] In a specific implementation scenario, after the first query feature is initialized, the first query feature and the multimodal feature can be processed based on the first self-attention mechanism to obtain the second query feature, and the self-attention mask of the first self-attention mechanism only allows the first query feature to unidirectionally perceive the multimodal feature. Figure 2 , Figure 2 is a schematic diagram of an embodiment of the first self-attention mechanism. Figure 2 As shown, illustratively, taking the initialization of N first query features as an example, if the multimodal features specifically include M different modal features, the M different modal features (such as Figure 2 The squares filled with diagonal lines in the figure represent) and N first query features (such as Figure 2 The square filled with dotted shadows in the middle represents the input information of the first self-attention mechanism, so that the correlation between any two features can be constructed based on the first self-attention mechanism, such as the first modal feature and the first modal feature, the first modal feature and the second feature modality, and so on. Examples are not given here one by one. Then, based on the correlation between any two features and the self-attention mask containing the attention mask value between any two features, feature weighting can be performed to obtain M different modal features and N first query features after being weighted by attention, and the N first query features after being weighted by attention can be used as N second query features. Figure 2The square filled with horizontal shadows in the middle represents the attention mask value representation masked, such as the value 0, and the square without shadows represents the attention mask value representation unmasked, such as the value 1, so the M modal features cannot perceive any first query feature, and the N first query features all perceive the M modal features, so that only the first query feature is allowed to perceive the multimodal feature unidirectionally. Of course, the above example is only a possible example of the first self-attention mechanism, and other possible implementation methods are not limited thereto. In addition, multimodal features can also be extracted based on the self-attention mechanism (for details, please refer to the above-mentioned related description). In order to facilitate the distinction from the above-mentioned first self-attention mechanism, it can be called the second self-attention mechanism, and the second self-attention mechanism and the first self-attention mechanism can share weight parameters. In the above method, the first query feature and the multimodal feature are processed based on the first self-attention mechanism to obtain the second query feature, and the self-attention mask of the first self-attention mechanism only allows the first query feature to unidirectionally perceive the multimodal feature. It is possible to extract only emotion-related features through the first query feature as much as possible to participate in subsequent emotion recognition, and to ensure the independence of the emotion inducement feature itself as much as possible.
[0043] In a specific implementation scenario, after obtaining the second query feature, the target emotion feature can be obtained based on the second query feature and the behavior action feature. As a possible example, fusion operations such as splicing, weighting, and averaging can be performed based on the second query feature and the behavior action feature to obtain the target emotion feature; or, as another possible example, the second query feature and the behavior action feature can also be processed based on the cross-attention mechanism to obtain the target emotion feature. Exemplarily, the Q feature of the cross-attention mechanism can be obtained based on the second query feature, and the K feature and V feature of the cross-attention mechanism can be obtained based on the behavior action feature, and then the Q feature, K feature, and V feature are processed to obtain the target emotion feature; or, as another possible example, after the aforementioned cross-attention mechanism is used, deep features can be extracted based on the feedforward neural network as the target emotion feature. Of course, the above examples are only several possible examples based on the second query feature and the behavior action feature in the actual application process, and are not limited here.
[0044] Step S14: Based on the target emotion feature, predict the target emotion of the target object.
[0045] In an implementation scenario, after obtaining the target emotion feature, the target emotion of the target object can be predicted based on the target emotion feature. Figure 3 , Figure 3 Schematic diagram of the framework of an embodiment of an emotion recognition model. Figure 2As shown, as a possible example, the target emotion can be identified by the emotion recognition model, and the emotion recognition model can include a first extraction network for extracting action behavior features (e.g., it can include the aforementioned video encoder and audio encoder), a second extraction network for extracting multimodal features (e.g., it can include the aforementioned ImageBind and a Transformer structure composed of a second self-attention mechanism and a feedforward neural network), and an emotion prediction network for predicting the target emotion based at least on the action behavior features and the multimodal features, such as specifically including an interactive subnetwork for combining the action behavior features and the multimodal features to obtain the target emotion features (e.g., it can include the aforementioned first self-attention mechanism, cross attention mechanism and feedforward neural network), and a prediction subnetwork for predicting emotions based on the target emotion features (e.g., it can include a fully connected layer, softmax, etc.). It should be noted that the above example is only a possible example of an emotion recognition model, and the specific structure of the emotion recognition model is not limited here. In this case, the target emotion features can be input into the prediction subnetwork, so that the target emotion of the target object (e.g., sadness, happiness, peace, anger, etc.) can be obtained. Alternatively, in the experimental stage using sample data, reference emotion features of several standard emotions can be pre-clustered through the sample data, and the correlation between the target emotion feature and the reference emotion features of each standard emotion can be measured again, so that at least one standard emotion can be selected as the target emotion of the target object based on the correlation. Of course, the above examples are only several possible examples of emotion prediction based on target emotion features, and do not limit the specific implementation method of emotion prediction based on target emotion features.
[0046] In an implementation scenario, taking the target emotion identified by the emotion recognition model as an example, in order to improve the model performance of the emotion recognition model, the emotion recognition model can be pre-trained based on the first sample data, and then trained based on the second sample data. It should be noted that the first sample data and the second sample data can both include sample behavior action data and sample multimodal data, the first sample data can cover two types of data, sample positive data and sample negative data, the annotation mark of the sample positive data can represent that the sample behavior action data and the sample multimodal data actually have an emotional causal relationship, the annotation mark of the sample negative data can represent that the sample behavior action data and the sample multimodal data actually do not have an emotional causal relationship, and the second sample data can be annotated with the sample emotion. Still taking the driving scene as an example, the sample behavior action data in the sample positive data may include but is not limited to video data of the driver's happy expression and audio data of the driver humming a song, and the sample multimodal data in the sample positive data may include but is not limited to image data of the flat land in front of the vehicle, audio data containing wind noise and tire noise, and instruction data of the driver requesting to play music, etc.; or, the sample behavior action data in the sample negative data may include but is not limited to video data of the driver's happy expression and audio data of the driver humming a song, and the sample multimodal data in the sample negative data may include but is not limited to image data of long-distance congestion in front of the vehicle, audio data containing horns one after another and cursing of drivers and passengers of surrounding vehicles, etc. Of course, the above examples are just a few possible examples of organizing sample data and sample negative data, and other scenarios can be deduced by analogy, and no more examples are given here. The above method first pre-trains with sample positive data and sample negative data, so that the emotion recognition model can learn the causal relationship between emotional inducements and behavioral actions, and then trains based on the second sample data labeled with sample emotions, so that after learning the causal relationship, the emotion recognition model can further combine the learned causal relationship to predict emotions, and then make progress step by step in the training process, which helps to improve the efficiency and effectiveness of model learning.
[0047] In a specific implementation scenario, the sample positive data and the sample negative data can be obtained by manual construction. Alternatively, the sample positive data and the sample negative data can also be automatically constructed based on behavioral action datasets such as IEMOCAP, CMU-MOSI, CMU-MOSEI, and multimodal datasets such as EMOTIC, Emotion6, SE20K8, UnBiasedEmo, DeepEmotion, VGAF, etc. It should be noted that the sample behavioral action data in the behavioral action dataset has labeled emotions, and the multimodal dataset contains a number of sample multimodal data. Exemplarily, at least one emotion keyword can be selected from the emotion keyword table to obtain the matching emotion of the sample multimodal data, and then the correlation between the labeled emotion and the matching emotion can be measured. For example, the emotion keyword table provided by StockEmotion can be used to measure the correlation between the labeled categories of the speaker multimodal emotion recognition dataset and the corresponding emotion keywords or labels of the picture or video. Please refer to Figure 4 , Figure 4 is a schematic diagram for constructing the first sample data. Figure 4 As shown, on this basis, based on the correlation between the labeled emotion and the matching emotion, the sample behavior action data and the sample multimodal data can be bound as sample positive data or sample negative data. For example, when the correlation is higher than a certain set threshold, the sample behavior action data and the sample multimodal data can be bound as sample positive data, and when the correlation is lower than a certain set threshold, the sample behavior action data and the sample multimodal data can be bound as sample negative data. Figure 4 As shown in the middle left figure, "I'm too tired" can match the multimodal data containing image data (tired expression) and audio data, so it can be bound as sample positive data. On the contrary, "one plus one equals two" cannot match the multimodal data containing image data (tired expression) and audio data, so it can be bound as sample negative data. In addition, in order to further expand on the basis of the existing modality, the text modality corresponding to the image or video can be generated with the help of the image or video caption model, and extended to the corresponding audio modality with the help of ImageBind. In the above method, a behavior action data set and a multimodal data set are obtained; wherein, the sample behavior action data in the behavior action data set has annotated emotions, and the multimodal data set contains a number of sample multimodal data, and then at least one emotion keyword is selected in the emotion keyword table to obtain the matching emotion of the sample multimodal data, so as to bind the sample behavior action data and the sample multimodal data as sample positive data or sample negative data based on the correlation between the annotated emotion and the matching emotion, and then the sample positive data and the sample negative data can be automatically constructed, which helps to reduce the difficulty of sample data collection as much as possible and improve the accuracy of data construction.
[0048] In a specific implementation scenario, after obtaining the first sample data, pre-training can be performed based on the first sample data. Specifically, the first sample behavior and action features can be extracted based on the sample behavior and action data, and the first sample multimodal features can be extracted based on the sample multimodal data, and then the first sample target emotion features can be obtained by fusing at least the first sample behavior and action features and the first sample multimodal features. It should be noted that the specific process of extracting the first sample behavior and action features can refer to the aforementioned description of extracting the behavior and action features, the specific process of extracting the first sample multimodal features can refer to the aforementioned description of extracting the multimodal features, and the specific process of obtaining the first sample target emotion features can refer to the aforementioned description of obtaining the target emotion features, which will not be repeated here. On this basis, predictions can be made based on the first sample target emotion features to obtain prediction tags, and the prediction tags characterize whether there is an emotional causal relationship between the sample behavior and action data and the sample multimodal data predictions. Exemplarily, in the following example, Figure 3 In the emotion recognition model shown, the emotion prediction network is then connected to the fully connected layer to perform classification prediction, thereby obtaining a prediction tag. Of course, the above examples are only possible examples in the actual application process, and other possible implementation methods are not limited thereto. After obtaining the prediction tag, the network parameters of the emotion recognition model can be adjusted based on at least the difference between the prediction tag and the annotation tag of the first sample data. Exemplarily, a loss function such as cross entropy can be used to measure the difference between the prediction tag and the annotation tag. In the above manner, by adjusting the network parameters of the emotion recognition model based on the difference between the prediction tag and the annotation tag, the emotion recognition model can be forced to learn the causal effect of the emotion inducement reflected in the multimodal data on the behavior action during the pre-training process, and combined with the sample positive data and the sample negative data for training, it can ensure the consistency of the causal effect of the multimodal emotion inducement on the behavior signal as much as possible. In addition, before adjusting the network parameters, the first sub-loss can also be obtained based on the similarity between the feature information of the text modality in the first sample multimodal feature and the feature information of various reference modalities. Exemplarily, the similarity between the feature information of the text modality and the feature information of various reference modalities can be measured by such as cosine similarity. Taking the reference modality including audio modality, video modality, and image modality as an example, the first sub-loss can be expressed as d(f t ,f a )+d(f t ,f v )+d(f t ,f i ), where d represents a similarity metric function such as cosine similarity, and f t Represents the feature information of the text modality, f a Represents the characteristic information of the audio mode, f v Represents the feature information of the video modality, fi Represents the characteristic information of the image modality. Other situations can be deduced by analogy, and no examples are given here one by one. On this basis, the second sub-loss can be obtained based on the difference between the predicted label and the labeled label of the first sample data, so that the network parameters of the emotion recognition model can be adjusted based on the first sub-loss and the second sub-loss. It should be noted that the total loss of pre-training can be negatively correlated with the first sub-loss and positively correlated with the second sub-loss, so that by constraining the minimization of the total loss, the first sub-loss can be forced to be as large as possible, that is, the same group of different modal emotion-inducing features are as consistent as possible, and the second sub-loss is forced to be as small as possible, that is, the emotional causal effect is predicted as accurately as possible. The above method, combined with the first sub-loss and the second sub-loss, adjusts the network parameters, which can further constrain the same group of different modal emotion-inducing features to be as consistent as possible.
[0049] In a specific implementation scenario, after the pre-training is completed, training can be performed again based on the second sample data. Specifically, the second sample behavior action features can be extracted based on the sample behavior action data, and the second sample multimodal features can be extracted based on the sample multimodal data, and then at least the second sample behavior action features and the second sample multimodal features are fused to obtain the second sample target emotion features. It should be noted that the specific process of extracting the second sample behavior action features can refer to the relevant description of extracting the behavior action features, the specific process of extracting the second sample multimodal features can refer to the relevant description of extracting the multimodal features, and the specific process of extracting the second sample target emotion features can refer to the relevant description of extracting the target emotion features, which will not be repeated here. On this basis, prediction can be made based on the second sample target emotion features to obtain predicted emotions, and then based on the difference between the predicted emotions and the sample emotions, the network parameters of the emotion prediction network are adjusted, and the parameters of the first extraction network and the second extraction network are kept fixed. For example, the difference between the predicted emotions and the sample emotions can be measured by a loss function such as cross entropy, and the parameters can be adjusted based on the loss value obtained by the measurement. That is to say, when starting training based on the second sample data, the network parameters of the emotion recognition model are initialized to the network parameters at the end of pre-training, and during the training process, the parameters of the first extraction network and the second extraction network are frozen so that the network parameters of the two remain unchanged, and only the network parameters of the emotion prediction network are adjusted. This enables the emotion prediction network to further improve the accuracy of emotion classification prediction on the basis of predicting causal relationships.
[0050] The above scheme obtains the behavior data of the target object and the multimodal data of the scene where the target object is located, and then extracts the behavior features based on the behavior data, and extracts the multimodal features based on the multimodal data, so as to obtain the target emotion features based on at least the behavior features and the multimodal features, and then predict the target emotion of the target object based on the target emotion features. Therefore, in the process of emotion recognition, not only the behavior data of the target object is taken into consideration, but also the scene where the target object is located is further taken into consideration, which can fully consider the induction factors of the target object's environment on the target object's emotions, and the scene where the target object is located is represented by multimodal data, which can express the induction factors as richly as possible from different modal angles, and then integrate the behavior features and multimodal features, which can fully model the emotional impact of the scene where the target object is located as an induction factor on the behavior of the target object. Therefore, the accuracy and applicability of emotion recognition can be improved.
[0051] See also Figure 5 , Figure 5 It is a schematic diagram of the framework of an embodiment of the emotion recognition device 50 of the present application. The emotion recognition device 50 includes: a data acquisition module 51, a feature extraction module 52, a feature fusion module 53 and an emotion prediction module 54. The data acquisition module 51 is used to obtain the behavior action data of the target object and obtain the multimodal data of the scene in which the target object is located; the feature extraction module 52 is used to extract the behavior action features based on the behavior action data and extract the multimodal features based on the multimodal data; the feature fusion module 53 is used to fuse at least the behavior action features and the multimodal features to obtain the target emotion features; the emotion prediction module 54 is used to predict the target emotion of the target object based on the target emotion features.
[0052] In the above scheme, the emotion recognition device 50 obtains the behavior data of the target object and the multimodal data of the scene in which the target object is located, and then extracts the behavior features based on the behavior data, and extracts the multimodal features based on the multimodal data, thereby at least fusing the behavior features and the multimodal features to obtain the target emotion features, and then predicting the target emotion of the target object based on the target emotion features. Therefore, in the emotion recognition process, not only the behavior data of the target object is taken into consideration, but also the scene in which the target object is located is further taken into consideration, which can fully consider the induction factors of the target object's environment on the target object's emotions, and the scene in which the target object is located is represented by multimodal data, which can express the induction factors as richly as possible from different modal angles, and then fuse the behavior features and the multimodal features, which can fully model the emotional impact of the scene in which the target object is located as an induction factor on the behavior of the target object. Therefore, the accuracy and applicability of emotion recognition can be improved.
[0053] In some disclosed embodiments, the feature fusion module 53 includes a feature initialization submodule for initializing a first query feature; the feature fusion module 53 includes an inducement extraction submodule for extracting feature information related to emotional inducements from multimodal features based on the first query feature to obtain a second query feature; the feature fusion module 53 includes a feature determination submodule for obtaining a target emotional feature based on the second query feature and behavioral action features; wherein the second query feature contains feature information related to emotional inducements in the multimodal features.
[0054] In some disclosed embodiments, the inducement extraction submodule is used to process the first query feature and the multimodal feature based on the first self-attention mechanism to obtain the second query feature; wherein the self-attention mask of the first self-attention mechanism only allows the first query feature to unidirectionally perceive the multimodal feature.
[0055] In some disclosed embodiments, multimodal features are extracted based on a second self-attention mechanism, and the second self-attention mechanism shares weight parameters with the first self-attention mechanism.
[0056] In some disclosed embodiments, the determination submodule is used to process the second query feature and the behavioral action feature based on a cross-attention mechanism to obtain a target emotion feature.
[0057] In some disclosed embodiments, a target emotion is identified by an emotion recognition model, which is pre-trained based on first sample data and then trained based on second sample data; wherein both the first sample data and the second sample data include sample behavior action data and sample multimodal data, and the first sample data covers two types of data, sample positive data and sample negative data, and the annotation mark of the sample positive data represents that there is actually an emotional causal relationship between the sample behavior action data and the sample multimodal data, and the annotation mark of the sample negative data represents that there is actually no emotional causal relationship between the sample behavior action data and the sample multimodal data, and the second sample data is annotated with the sample emotion.
[0058] In some disclosed embodiments, the emotion recognition device 50 includes a first sample extraction module, which is used to extract first sample behavior and action features based on sample behavior and action data, and extract first sample multimodal features based on sample multimodal data; the emotion recognition device 50 includes a first sample fusion module, which is used to fuse at least the first sample behavior and action features and the first sample multimodal features to obtain first sample target emotion features; the emotion recognition device 50 includes a first sample prediction module, which is used to predict based on the first sample target emotion features to obtain a prediction label; wherein the prediction label characterizes whether there is an emotional causal relationship between the sample behavior and action data and the sample multimodal data prediction; the emotion recognition device 50 includes a first sample adjustment module, which is used to adjust the network parameters of the emotion recognition model based on at least the difference between the prediction label and the labeled label of the first sample data.
[0059] In some disclosed embodiments, the sample multimodal data involves a text modality and at least one reference modality, and the emotion recognition device 50 includes a training loss measurement module for obtaining a first sub-loss based on the similarity between feature information of the text modality in the first sample multimodal feature and feature information of various reference modalities; the first sample adjustment module is specifically used to obtain a second sub-loss based on the difference between the predicted label and the labeled label of the first sample data; based on the first sub-loss and the second sub-loss, the network parameters of the emotion recognition model are adjusted.
[0060] In some disclosed embodiments, the emotion recognition device 50 includes a set acquisition module for acquiring a behavior action data set and a multimodal data set; wherein the sample behavior action data in the behavior action data set has labeled emotions, and the multimodal data set contains a number of sample multimodal data; the emotion recognition device 50 includes an emotion matching module for selecting at least one emotion keyword in an emotion keyword table to obtain a matching emotion for the sample multimodal data; the emotion recognition device 50 includes a data binding module for binding the sample behavior action data and the sample multimodal data as sample positive data or sample negative data based on the correlation between the labeled emotions and the matching emotions.
[0061] In some disclosed embodiments, the emotion recognition model includes a first extraction network for extracting behavioral action features, a second extraction network for extracting multimodal features, and an emotion prediction network for predicting a target emotion based at least on the behavioral action features and the multimodal features. The emotion recognition device 50 includes a second sample extraction module for extracting second sample behavioral action features based on sample behavioral action data, and extracting second sample multimodal features based on sample multimodal data; the emotion recognition device 50 includes a second sample fusion module for fusing at least the second sample behavioral action features and the second sample multimodal features to obtain a second sample target emotion feature; the emotion recognition device 50 includes a second sample prediction module for predicting based on the second sample target emotion feature to obtain a predicted emotion; the emotion recognition device 50 includes a second sample adjustment module for adjusting network parameters of the emotion prediction network based on the difference between the predicted emotion and the sample emotion, and keeping the parameters of the first extraction network and the second extraction network fixed.
[0062] In some disclosed embodiments, the feature extraction module 52 includes a video encoding submodule for performing video encoding based on the video data recorded for the target object in the behavioral action data to obtain a video encoding representation; the feature extraction module 52 includes an audio encoding submodule for performing audio encoding based on the audio data collected for the target object in the behavioral action data to obtain an audio encoding representation; the feature extraction module 52 includes an audio and video fusion submodule for performing fusion based on the video encoding representation and the audio encoding representation to obtain behavioral action features.
[0063] In some disclosed embodiments, the feature extraction module 52 includes an initial extraction submodule, which is used to extract the initial feature representation of each modality in the same semantic space based on each modal data in the multimodal data; the feature extraction module 52 includes a self-attention processing submodule, which is used to process the initial feature representation of each modality based on at least a second self-attention mechanism to obtain multimodal features.
[0064] In some disclosed embodiments, the behavioral action data includes at least one of the following: video data recorded of the target object, audio data collected of the target object; and / or, the multimodal data includes at least one of the following: image data taken of the scene where the target object is located, audio data of the environment where the target object is located, and demand data input by the target object in the environment where it is located.
[0065] See also Figure 6 , Figure 6 6 is a schematic diagram of a framework of an embodiment of an electronic device 60 of the present application. The electronic device 60 includes a memory 61 and a processor 62 coupled to each other, the memory 61 stores program instructions, and the processor 62 is used to execute the program instructions to implement the steps in any of the above-mentioned emotion recognition method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here.
[0066] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps in any of the above-mentioned emotion recognition method embodiments. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.
[0067] In the above scheme, the electronic device 60 obtains the behavior data of the target object, and obtains the multimodal data of the scene in which the target object is located, and then extracts the behavior features based on the behavior data, and extracts the multimodal features based on the multimodal data, so as to obtain the target emotion features based on at least the behavior features and the multimodal features, and then predict the target emotion of the target object based on the target emotion features. Therefore, in the process of emotion recognition, not only the behavior data of the target object is taken into consideration, but also the scene in which the target object is located is further taken into consideration, which can fully consider the induction factors of the target object's environment on the target object's emotions, and the scene in which the target object is located is represented by multimodal data, which can express the induction factors as richly as possible from different modal angles, and then integrate the behavior features and the multimodal features, which can fully model the emotional impact of the scene in which the target object is located as an induction factor on the behavior of the target object. Therefore, the accuracy and applicability of emotion recognition can be improved.
[0068] See also Figure 7 , Figure 7 It is a schematic diagram of the framework of an embodiment of the smart cockpit 70 of the present application. The smart cockpit 70 at least includes a collection device 71 and a control device 72 that is communicatively connected to the collection device 71. The collection device 71 is used to collect the behavior and action data of the target object and the multimodal data of the environment in which the target object is located. The control device 72 is an electronic device in the above-mentioned electronic device embodiment. For details, please refer to the above-mentioned electronic device embodiment, which will not be repeated here.
[0069] In one implementation scenario, the acquisition device 71 may include, but is not limited to: a sound pickup device (not shown) for acquiring audio data, a camera device (not shown) for recording video data, etc., which is not limited here.
[0070] In one implementation scenario, the control device 72 may specifically include but is not limited to a vehicle computer (not shown), etc., which is not limited here.
[0071] The above scheme at least includes a collection device 71 and a control device 72 that is communicatively connected to the collection device 71. The collection device 71 is used to collect the behavior and action data of the target object and the multimodal data of the target object's environment. The control device 72 is the electronic device in the above electronic device embodiment. Therefore, in the process of emotion recognition, not only the behavior and action data of the target object are taken into consideration, but also the scene in which the target object is located is further taken into consideration, which can fully consider the inducing factors of the target object's environment on the target object's emotions, and the scene in which the target object is located is represented by multimodal data, which can express the inducing factors as richly as possible from different modal perspectives, and then integrate the behavior and action features with the multimodal features, which can fully model the emotional impact of the scene in which the target object is located as an inducing factor on the target object's behavior and actions. Therefore, the accuracy and applicability of emotion recognition can be improved.
[0072] See also Figure 8 , Figure 8 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 80 of the present application. The computer-readable storage medium 80 stores program instructions 81 that can be executed by a processor, and the program instructions 81 are used to implement the steps in any of the above-mentioned emotion recognition method embodiments.
[0073] In the above scheme, the computer-readable storage medium 80 obtains the behavior data of the target object and obtains the multimodal data of the scene in which the target object is located, and then extracts the behavior features based on the behavior data, and extracts the multimodal features based on the multimodal data, thereby at least integrating the behavior features and the multimodal features to obtain the target emotion features, and then predicting the target emotion of the target object based on the target emotion features. Therefore, in the process of emotion recognition, not only the behavior data of the target object is taken into consideration, but also the scene in which the target object is located is further taken into consideration, which can fully consider the induction factors of the target object's environment on the target object's emotions, and the scene in which the target object is located is characterized by multimodal data, which can express the induction factors as richly as possible from different modal perspectives, and then integrate the behavior features and the multimodal features, which can fully model the emotional impact of the scene in which the target object is located as an induction factor on the behavior of the target object. Therefore, the accuracy and applicability of emotion recognition can be improved.
[0074] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0075] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.
[0076] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0077] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0078] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0079] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.
[0080] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.
Claims
1. An emotion recognition method, characterized in that: include: Acquire the behavior data of the target object, and acquire the multimodal data of the scene in which the target object is located; Extracting behavioral action features based on the behavioral action data, and extracting multimodal features based on the multimodal data; At least based on the behavior action feature and the multimodal feature, a fusion is performed to obtain a target emotion feature; Based on the target emotion feature, the target emotion of the target object is predicted.
2. The method according to claim 1, characterized in that The step of fusing at least the behavior action feature and the multimodal feature to obtain a target emotion feature includes: Initialize and obtain the first query feature; Based on the first query feature, extracting feature information related to emotion inducements from the multimodal feature to obtain a second query feature; Based on the second query feature and the behavior action feature, the target emotion feature is obtained; wherein the second query feature includes feature information related to emotion inducements in the multimodal feature.
3. The method according to claim 2, characterized in that The step of extracting feature information related to emotion inducements from the multimodal features based on the first query feature to obtain a second query feature includes: Processing the first query feature and the multimodal feature based on a first self-attention mechanism to obtain the second query feature; Among them, the self-attention mask of the first self-attention mechanism only allows the first query feature to unidirectionally perceive the multimodal feature.
4. The method according to claim 3, characterized in that The multimodal features are extracted based on a second self-attention mechanism, and the second self-attention mechanism shares weight parameters with the first self-attention mechanism.
5. The method according to claim 2, characterized in that: The obtaining the target emotion feature based on the second query feature and the behavior action feature includes: The second query feature and the behavior action feature are processed based on a cross attention mechanism to obtain the target emotion feature.
6. The method according to claim 1, characterized in that The target emotion is identified by an emotion recognition model, wherein the emotion recognition model is pre-trained based on the first sample data and then trained based on the second sample data; Among them, the first sample data and the second sample data both include sample behavior action data and sample multimodal data, the first sample data covers two types of data, sample positive data and sample negative data, the annotation mark of the sample positive data represents that there is actually an emotional causal relationship between the sample behavior action data and the sample multimodal data, the annotation mark of the sample negative data represents that there is actually no emotional causal relationship between the sample behavior action data and the sample multimodal data, and the second sample data is annotated with sample emotions.
7. The method according to claim 6, characterized in that The step of training the emotion recognition model based on the first sample data comprises: Extracting a first sample behavior action feature based on the sample behavior action data, and extracting a first sample multimodal feature based on the sample multimodal data; Obtaining a first sample target emotion feature by fusing at least the first sample behavior action feature and the first sample multimodal feature; Predicting based on the first sample target emotion feature to obtain a prediction mark; wherein the prediction mark indicates whether there is an emotion causal relationship between the sample behavior action data and the sample multimodal data prediction; Based at least on the difference between the predicted label and the labeled label of the first sample data, the network parameters of the emotion recognition model are adjusted.
8. The method according to claim 7, characterized in that The sample multimodal data involves a text modality and at least one reference modality. Before adjusting the network parameters of the emotion recognition model based at least on the difference between the predicted tag and the labeled tag of the first sample data, the method further includes: Obtaining a first sub-loss based on the similarity between the feature information of the text modality in the first sample multimodal feature and the feature information of various reference modalities; The adjusting the network parameters of the emotion recognition model based at least on the difference between the predicted label and the labeled label of the first sample data includes: Obtaining a second sub-loss based on a difference between the predicted label and the labeled label of the first sample data; Based on the first sub-loss and the second sub-loss, a network parameter of the emotion recognition model is adjusted.
9. The method according to claim 6, characterized in that The step of constructing the first sample data includes: Acquire a behavior action data set and a multimodal data set; wherein the sample behavior action data in the behavior action data set has annotated emotions, and the multimodal data set includes a plurality of sample multimodal data; Select at least one emotion keyword from the emotion keyword table to obtain a matching emotion of the sample multimodal data; Based on the correlation between the labeled emotion and the matching emotion, the sample behavior action data and the sample multimodal data are bound as the sample positive data or the sample negative data.
10. The method according to claim 6, characterized in that The emotion recognition model includes a first extraction network for extracting the behavior action features, a second extraction network for extracting the multimodal features, and an emotion prediction network for predicting the target emotion based on at least the behavior action features and the multimodal features, and the step of training based on the second sample data includes: Extracting a second sample behavior action feature based on the sample behavior action data, and extracting a second sample multimodal feature based on the sample multimodal data; Obtaining a second sample target emotion feature by fusing at least the second sample behavior action feature and the second sample multimodal feature; Performing prediction based on the target emotion feature of the second sample to obtain a predicted emotion; Based on the difference between the predicted emotion and the sample emotion, the network parameters of the emotion prediction network are adjusted, and the parameters of the first extraction network and the second extraction network are kept fixed.
11. The method according to claim 1, characterized in that: The extracting behavior action features based on the behavior action data includes: Performing video encoding on the video data recorded by the target object in the behavior action data to obtain a video encoding representation, and performing audio encoding on the audio data collected by the target object in the behavior action data to obtain an audio encoding representation; The behavior action feature is obtained by fusing the video coding representation and the audio coding representation.
12. The method according to claim 1, characterized in that The extracting multimodal features based on the multimodal data includes: Based on each modal data in the multimodal data, extracting initial feature representations of each modality in the same semantic space; The initial feature representations of the respective modalities are processed at least based on a second self-attention mechanism to obtain the multimodal features.
13. The method according to any one of claims 1 to 12, characterized in that: The behavior data includes at least one of the following: video data recorded on the target object, audio data collected on the target object; And / or, the multimodal data includes at least one of the following: image data captured at the scene where the target object is located, audio data of the environment where the target object is located, and demand data input by the target object in the environment where the target object is located.
14. An emotion recognition device, characterized in that: include: A data acquisition module is used to acquire the behavior data of the target object and acquire the multimodal data of the scene in which the target object is located; A feature extraction module, used to extract behavior action features based on the behavior action data, and to extract multimodal features based on the multimodal data; A feature fusion module, configured to fuse at least the behavior action feature and the multimodal feature to obtain a target emotion feature; The emotion prediction module is used to predict the target emotion of the target object based on the target emotion feature.
15. An electronic device, characterized in that: The device at least comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the emotion recognition method according to any one of claims 1 to 13.
16. A smart cockpit, characterized in that: It at least includes a collection device and a control device communicatively connected to the collection device, the collection device is used to collect behavioral action data of the target object and multimodal data of the environment in which the target object is located, and the control device is the electronic device according to claim 14.
17. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the emotion recognition method described in any one of claims 1 to 13.
Citation Information
Cited By
Scene-adaptive digital human behavior prediction method and device, and storage medium
CN119992655A
A scene-adaptive digital human behavior prediction method, device and storage medium
CN119992655B
Data processing method, target container determination method, device, equipment, medium and product
CN120744860A
Vehicle service processing method and device, computer equipment and readable storage medium
CN121217789A
