Methods, devices, and electronic equipment for analyzing object emotions.

Abstract

This invention provides a method, apparatus, and electronic device for analyzing the emotion of an object. The method includes: extracting static facial features and dynamic features from multimedia data associated with the target object; the dynamic features include one or more of facial expression change features, vocal features, and linguistic content features; inputting the static facial features and dynamic features into a pre-trained object emotion analysis model; and performing feature fusion processing on the static facial features and dynamic features through the object emotion analysis model to output the emotion analysis result. This method performs feature fusion processing on static facial features and dynamic features. Since dynamic features also contain feature information representing emotion, combining static facial features with dynamic features for emotion analysis can, to some extent, reduce the influence of interfering features in static facial features on the emotion analysis result, strengthen the role of feature information representing emotion, and thus improve the accuracy of the emotion analysis result.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, and electronic device for analyzing the emotions of an object. Background Technology

[0002] In related technologies, facial emotion analysis of faces or other objects can be performed using various methods such as facial action units, basic emotion type detection, and arousal-stimulation analysis. These methods require extracting facial features from images and then using deep learning models to perform emotion analysis on these features to obtain the emotion analysis results. However, facial features contain various interfering factors, such as lighting, occlusion, and personalized facial features. These factors can affect how deep learning models learn or extract emotion-related features from facial features, leading to lower accuracy in the emotion analysis results output by the deep learning models. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method, apparatus and electronic device for analyzing the emotions of an object, so as to reduce the influence of interfering features in static facial features on the emotion analysis results to a certain extent, strengthen the role of feature information representing emotions, and thus improve the accuracy of emotion analysis results.

[0004] In a first aspect, embodiments of the present invention provide a method for analyzing the emotion of an object, the method comprising: acquiring multimedia data associated with a target object; extracting static facial features and dynamic features of the target object from the multimedia data; wherein the dynamic features include one or more of the target object's facial expression change features, voice features, and language content features; inputting the static facial features and dynamic features into a pre-trained object emotion analysis model; performing feature fusion processing on the static facial features and dynamic features through the object emotion analysis model to obtain fused features; and outputting the emotion analysis result of the target object based on the fused features.

[0005] The steps described above for extracting dynamic features of a target object from multimedia data include: if the dynamic features include facial expression change features, obtaining a specified video frame sequence from the video data in the multimedia data; wherein, the facial expression of the target object changes dynamically in the video frame sequence; and extracting the facial expression change features of the target object from the video frame sequence using a pre-trained facial expression feature extraction model.

[0006] The steps described above for extracting dynamic features of a target object from multimedia data include: if the dynamic features include sound features, obtaining a specified audio sequence from the audio data in the multimedia data; wherein the audio sequence includes sound signals emitted by the target object; and extracting the sound features of the target object from the audio sequence using a pre-trained sound feature extraction model; wherein the sound features include one or more of frequency features, pitch features, volume features, and energy features.

[0007] The steps described above for extracting dynamic features of a target object from multimedia data include: if the dynamic features include language content features, obtaining the language content text of the target object from the subtitle data and / or audio data in the multimedia data; extracting the language content features of the target object from the language content text using a pre-trained language content feature extraction model; wherein, the language content features are used to: characterize the linguistic meaning of the language spoken by the target object.

[0008] The aforementioned object sentiment analysis model includes at least one dynamic feature processing module, which is used to input dynamic features of the corresponding type. The step of performing feature fusion processing on static facial features and dynamic features through the object sentiment analysis model to obtain fused features includes: inputting the dynamic features corresponding to the dynamic feature processing module and the static facial features of the target object into the dynamic feature processing module; performing feature fusion processing on the static facial features and dynamic features through the attention network in the dynamic feature extraction module to obtain fused features; wherein, the fused features include: common key features representing emotions between static facial features and dynamic features.

[0009] Before the step of inputting the dynamic features corresponding to the dynamic feature processing module and the static facial features of the target object into the dynamic feature processing module, the method further includes: inputting the dynamic features into the temporal model and outputting the context features of the dynamic features; wherein, the context features are used to: characterize the changing trend of the features in the dynamic features over time; adding temporal position information to the context features, and determining the context features with added temporal position information as the dynamic features to be input into the dynamic feature extraction module; wherein, the temporal position information is used to: indicate the sequential position of the features in the context features over time.

[0010] The attention network in the aforementioned dynamic feature processing module includes a self-attention network and a cross-attention network. The step of performing feature fusion processing on static facial features and dynamic features through the attention network in the dynamic feature extraction module to obtain fused features includes: transforming the dynamic features to obtain the first input parameters of the self-attention network, inputting the first input parameters into the self-attention network, and outputting intermediate features of the dynamic features; wherein, the intermediate features are used to: characterize the autocorrelation of the dynamic features; determining the second input parameters of the cross-attention network based on the intermediate features, determining the third input parameters of the cross-attention network based on the static facial features, inputting the second input parameters and the third input parameters into the cross-attention network to obtain the output result, and determining the fused features based on the output result.

[0011] The steps for determining the second input parameters of the cross-attention network based on intermediate features include: performing a first fusion process on the intermediate features and dynamic features to obtain a first fusion result; and performing a transformation process on the first fusion result to obtain the second input parameters of the cross-attention network.

[0012] The steps for determining the third input parameters of the cross-attention network based on static facial features include: transforming the static facial features to obtain the third input parameters of the cross-attention network.

[0013] The steps for determining fusion features based on the output results include: performing a second fusion process on the first fusion result corresponding to the output result and the second input parameter to obtain a second fusion result; wherein the second input parameter is obtained by transforming the first fusion result; inputting the second fusion result into a preset first multilayer perceptron, and performing a mapping process on the second fusion result through the first multilayer perceptron to obtain a mapping result; and performing a third fusion process on the mapping result and the second fusion result to obtain fusion features.

[0014] The aforementioned object sentiment analysis model includes multiple dynamic feature processing modules. The dynamic feature processing modules are used to input dynamic features of the corresponding type and output fused features corresponding to the dynamic features. The steps of outputting the sentiment analysis results of the target object based on the fused features include: splicing the fused features output by the dynamic feature processing modules and the static facial features to obtain spliced ​​features; inputting the spliced ​​features into a preset second multilayer perceptron and outputting the sentiment analysis results of the target object.

[0015] The aforementioned second multilayer perceptron includes multiple branch networks. The step of inputting the spliced ​​features into a preset second multilayer perceptron and outputting the emotion analysis result of the target object includes: inputting the spliced ​​features into multiple branch networks of the second multilayer perceptron respectively; wherein, each branch network has a preset feature mapping method; the feature mapping method includes several of the following: linear combination mapping based on preset facial action units, linear combination mapping based on preset multiple basic emotion types, and linear representation mapping based on the positive / negative degree and intensity of the emotion; the spliced ​​features are mapped by the branch networks according to the feature mapping method corresponding to the branch networks to obtain the emotion analysis result output by the branch networks.

[0016] The aforementioned multimedia data refers to video data containing the target object within the video frame. The steps for extracting static facial features and dynamic features of the target object from the multimedia data include: taking each video frame in the video data as the current video frame, and performing the following operations on the current video frame: obtaining static facial features from the current video frame; obtaining facial expression change features from the video frame sequence containing the current video frame; obtaining sound features from the audio data corresponding to the video frame sequence; and obtaining language content features from the audio data and / or subtitle data corresponding to the video frame sequence.

[0017] Following the steps of outputting the sentiment analysis results of the target object based on the fusion features described above, the method further includes: obtaining the sentiment analysis results corresponding to video frames in the video data; determining, based on the sentiment analysis results corresponding to the video frames, that there are video frames with abrupt changes in sentiment in the video data; wherein, the sentiment analysis results of the video frames with abrupt changes in sentiment are the first analysis results, and the sentiment analysis results of the video frames before and after the video frames with abrupt changes in sentiment are the second analysis results; the first analysis results are different from the second analysis results; and based on the second analysis results, smoothing is performed on the sentiment analysis results of the video frames with abrupt changes in sentiment to obtain the smoothed sentiment analysis results.

[0018] Secondly, embodiments of the present invention provide an object emotion analysis device, the device comprising: a feature extraction module, used to acquire multimedia data associated with a target object, and extract static facial features and dynamic features of the target object from the multimedia data; wherein, the dynamic features include one or more of the target object's facial expression change features, voice features, and language content features; a result output module, used to input the static facial features and dynamic features into a pre-trained object emotion analysis model, perform feature fusion processing on the static facial features and dynamic features through the object emotion analysis model to obtain fused features; and output the emotion analysis result of the target object based on the fused features.

[0019] Thirdly, embodiments of the present invention provide an electronic device, including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the above-mentioned method for analyzing object emotions.

[0020] Fourthly, embodiments of the present invention provide a machine-readable storage medium storing machine-executable instructions. When the machine-executable instructions are invoked and executed by a processor, the machine-executable instructions cause the processor to implement the aforementioned method for analyzing object emotions.

[0021] The embodiments of the present invention bring the following beneficial effects:

[0022] The aforementioned method, apparatus, and electronic device for analyzing object emotions acquire multimedia data associated with the target object, extracting static facial features and dynamic features of the target object from the multimedia data. The dynamic features include one or more of the target object's facial expression changes, vocal features, and linguistic content features. The static facial features and dynamic features are input into a pre-trained object emotion analysis model, which performs feature fusion processing on the static facial features and dynamic features to obtain fused features. Based on the fused features, the emotion analysis result of the target object is output. In this approach, the object emotion analysis model performs feature fusion processing on the static facial features and dynamic features of the target object to output the emotion analysis result. Since dynamic features also contain information representing emotions, combining static facial features with dynamic features for emotion analysis can, to some extent, reduce the influence of interfering features in the static facial features on the emotion analysis result, strengthen the role of information representing emotions, and thus improve the accuracy of the emotion analysis result.

[0023] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0024] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0025] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating an object emotion analysis method provided in an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of the module structure and data flow of a dynamic feature processing module provided in an embodiment of the present invention;

[0028] Figure 3 A schematic diagram of the model structure and data flow of an object sentiment analysis model provided in an embodiment of the present invention;

[0029] Figure 4 A schematic diagram of the structure of an object emotion analysis device provided in an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] In related technologies, when performing sentiment analysis using facial action units (AUs), taking faces as an example, facial landmarks need to be introduced as local information to supervise the training of deep learning models. A common practice is to use facial landmarks as supervision information, allowing the deep learning model to complete multi-task training of facial landmarks and facial action units, resulting in a large learning burden on the model. Furthermore, since facial landmarks contain personalized identity information such as facial features, the model is prone to overfitting to the recognition of identity information, leading to lower accuracy in the sentiment analysis results output by the model.

[0033] When performing emotion analysis using basic emotion type detection, the results are easily affected by complex factors such as face occlusion and lighting. In addition, due to the large variance within basic emotion types and the similarity and ambiguity between different types, the emotion analysis results of models built on basic emotion types have a large degree of uncertainty.

[0034] When performing sentiment analysis using the Valence-Arousal (VA) method, it relies on data from the aforementioned facial action units. This can easily lead to the model overfitting to personalized identity information, resulting in lower accuracy of the sentiment analysis results.

[0035] Based on the above, embodiments of the present invention provide a method, apparatus, and electronic device for analyzing the emotions of objects. This technology can be applied to emotion analysis scenarios of human face objects, animal objects, or other objects with emotional fluctuations.

[0036] To facilitate understanding of this embodiment, a method for analyzing object emotions disclosed in this invention will first be described in detail, such as... Figure 1 As shown, the method includes the following steps:

[0037] Step S102: Acquire multimedia data associated with the target object, and extract static facial features and dynamic features of the target object from the multimedia data; wherein, the dynamic features include one or more of the target object's facial expression change features, voice features, and language content features;

[0038] The multimedia data can include data in various formats, such as video, images, audio, and text. This embodiment aims to analyze a target object; therefore, the multimedia data is typically associated with the target object. For example, the video may include the target object, the image may include the target object, the audio may include the target object's voice, the text may include the target object's speech, or other forms of output. The target object can be a person, animal, bionic robot, or other object exhibiting emotional fluctuations.

[0039] The static facial features of the target object can be extracted from image data containing the target object, which can also be video frame images. This static facial feature data can be obtained through a pre-trained facial feature extraction model, which can specifically consist of convolutional neural networks, residual networks, etc. These static facial features can characterize the target object's facial features, posture features, and expression features, and can be understood as a hybrid feature set. If the model is trained solely based on static facial features, it is difficult for the model to learn only the expression features; it may also learn the target object's facial features, causing the model to be influenced by the target object's facial features when analyzing expressions. Furthermore, the model may learn the target object's posture features, causing the model to be influenced by the target object's posture when analyzing expressions, thus reducing the accuracy of the model's expression analysis.

[0040] Based on this, this embodiment extracts dynamic features in addition to static facial features. These dynamic features are those that can reflect the target object's emotions to a certain extent. By using these dynamic features to assist static facial features, the model can learn key features representing emotions during training, thereby enabling the model to extract these key features during application and improving the accuracy of the model's emotion analysis results. Specifically, the specific types of dynamic features can be determined according to actual needs.

[0041] The aforementioned facial expression features can be obtained from video data. In video data, for the same target object, when facial expressions change, the shape, shape, and position of facial features all change. Facial expression features representing changes in the target object's expression across different video frames can be extracted from the video data. The aforementioned vocal features can be extracted from audio data. Under different emotions, the vocal features emitted by the same target object will also differ. For example, in a calm state, the voice is relatively soft; in a state of surprise, the voice may be relatively sharp; in a state of anger, the voice may be relatively low. Therefore, vocal features contain characteristic information representing emotions; these vocal features can specifically include frequency features, pitch features, volume features, and energy features. The aforementioned language content features can be extracted from audio data, subtitle data, or text typed out by the target object. For the same target object, the spoken content will also differ under different emotions. For example, in a happy state, the spoken content tends to be positive and sunny; in a sad state, the spoken content tends to be depressed and gloomy. Therefore, language content features also contain characteristic information representing emotions.

[0042] Step S104: Input static facial features and dynamic features into the pre-trained object sentiment analysis model, perform feature fusion processing on the static facial features and dynamic features through the object sentiment analysis model to obtain fused features; output the sentiment analysis results of the target object based on the fused features.

[0043] Considering that both static and dynamic facial features contain features representing emotions, in this embodiment, a feature fusion process is performed on static and dynamic facial features using an object emotion analysis model. Through feature fusion, the emotional information of the static and dynamic facial features is superimposed, further strengthening the features representing emotions and thus improving the accuracy of the emotion analysis results.

[0044] In practical implementation, the aforementioned object sentiment analysis model can be achieved through a combination of various network models, such as attention networks, multilayer perceptron networks, feature overlay networks, feature fusion networks, and temporal networks. During training, this model performs feature fusion processing on the static and dynamic facial features of the sample objects in the training samples. This strengthens the emotional expression information in both static and dynamic features, enabling the model to learn key features representing emotions and resulting in highly accurate sentiment analysis outputs.

[0045] The aforementioned method for analyzing object emotions involves acquiring multimedia data associated with the target object, extracting static and dynamic features from the multimedia data, and then considering dynamic features, including one or more of the target object's facial expression changes, vocal features, and linguistic content features. The static and dynamic features are input into a pre-trained object emotion analysis model, which performs feature fusion processing on the static and dynamic features to obtain fused features. Based on these fused features, the emotion analysis result for the target object is output. In this approach, the object emotion analysis model performs feature fusion processing on the static and dynamic features of the target object to output the emotion analysis result. Since dynamic features also contain information representing emotions, combining static and dynamic features for emotion analysis can, to some extent, reduce the influence of interfering features in static facial features on the emotion analysis result, strengthen the role of information representing emotions, and thus improve the accuracy of the emotion analysis result.

[0046] The following examples further illustrate the extraction methods for static facial features and each dynamic feature.

[0047] First, the aforementioned static facial features can be extracted using a pre-trained static feature extractor. The structure of this static feature extractor can be an Inception-ResNet structure, or other residual network structures or convolutional neural network structures.

[0048] If the aforementioned dynamic features include facial expression change features, a specified video frame sequence is obtained from the video data in the multimedia data; wherein, the facial expression of the target object changes dynamically in the video frame sequence; and the facial expression change features of the target object are extracted from the video frame sequence using a pre-trained facial expression feature extraction model.

[0049] The aforementioned video frame sequence can be a continuous sequence of video frames from video data. For example, for a laughing expression, this video frame sequence records a series of actions of the target object, from opening the mouth, widening the mouth, facial deformation, eye deformation, mouth shrinking, and finally closing the mouth. This video frame sequence may include, for example, 50 frames. In this video frame sequence, the target object's expression undergoes subtle changes in each frame. By connecting the video frames continuously, the dynamic changes in the target object's expression can be observed. The aforementioned expression feature extraction model can specifically be an expression encoding model or other types of feature extraction models. This expression feature extraction model can compress the initial high-dimensional information extracted from the video frame sequence into a low-dimensional manifold space. In this manifold space, the similarity features of expressions are preserved; that is, in this manifold space, if the spatial distance between two feature points is small, then the actual expressions corresponding to these two feature points are also similar. Facial expression feature extraction models typically use large-scale facial expression datasets as training samples for pre-training, allowing for the annotation of fine-grained facial expression features. Due to the large volume of training data, these datasets can encompass the expressions of various objects under different actions and postures. Therefore, facial expression feature extraction models can learn facial expression features independent of an object's appearance, actions, and posture. If dynamic features include facial expression change characteristics, it provides valuable prior information for subsequent sentiment analysis.

[0050] If the aforementioned dynamic features include sound features, a specified audio sequence is obtained from the audio data in the multimedia data; wherein the audio sequence includes the sound signal emitted by the target object; the sound features of the target object are extracted from the audio sequence using a pre-trained sound feature extraction model; wherein the sound features include one or more of frequency features, pitch features, volume features, and energy features.

[0051] The aforementioned sound signals can be recorded when the target object speaks, sings, or makes sounds without semantic meaning. The audio sequence is obtained by recording these sound signals. In one specific approach, the audio corresponding to the video frame sequence from which facial expression change features are extracted can be used as the aforementioned audio sequence. The aforementioned sound feature extraction model can be implemented using the open-source speech library librosa, or it can be implemented using other network structures or algorithms. The frequency features in the aforementioned sound features can specifically be MFCC (Mel Frequency Cepstrum Coefficient) features, which can also be frequency features in the Fourier or Laplace frequency domains; the energy features can specifically be short-time energy features, such as short-time energy average amplitude, short-time zero-crossing rate, and other feature parameters. Because sound features are relatively low-level features with strong generalization, the sound feature extraction model does not need to be trained on a dataset and is not easily affected by the data distribution in the dataset. Therefore, this sound feature does not carry personalized, identity-representing feature information; the feature information representing emotion accounts for a large proportion of this sound feature.

[0052] If the aforementioned dynamic features include language content features, the language content text of the target object is obtained from the subtitle data and / or audio data in the multimedia data; the language content features of the target object are extracted from the language content text using a pre-trained language content feature extraction model; wherein, the language content features are used to: characterize the linguistic meaning of the language spoken by the target object.

[0053] For subtitle data, which is typically in text format, this data records what the target subject said. Therefore, the target subject's spoken text can be directly extracted from the subtitle data. For audio data, speech recognition tools can be used to identify what the target subject said in the audio data, thus obtaining the spoken text in text format. In one specific approach, the target subject's spoken text can be extracted from the subtitle or audio data corresponding to the video frame sequence from which facial expression change features were previously extracted. In one example, the spoken text might be "Oh, my god," which typically contains features representing surprise.

[0054] The aforementioned language content feature extraction model is primarily used to identify the semantic features of the aforementioned language content text. It can be implemented using the text feature model BERT (Bidirectional Encoder Representation from Transformers) or other text semantic feature extraction models. The language content feature extraction model can be trained using a large corpus and can extract features between adjacent words. Since the language content feature extraction model extracts semantic features, representing the linguistic meaning of the language spoken by the target object, the language content features do not carry personalized, identity-related features; instead, features representing emotion constitute a significant portion of the language content features.

[0055] When the target is a human, emotions can be perceived through vision, hearing, and text. Based on this, feature data from these three modalities were extracted in the above embodiments. Considering that static facial features contain a large proportion of personalized information representing the identity of the subject, training the model using only static facial features would cause the model to overfit to personalized information. Adding multimodal dynamic features can alleviate the overfitting problem, prevent the model from learning personalized information, focus on learning emotion-related features, and improve the accuracy of the model in analyzing emotions.

[0056] The following embodiments further describe the model structure of the object sentiment analysis model and the specific implementation method of feature fusion.

[0057] The object sentiment analysis model includes at least one dynamic feature processing module, which is used to input dynamic features of the corresponding type. If the dynamic features include three types—facial expression features, vocal features, and linguistic content features—then the object sentiment analysis model includes three parallel dynamic feature processing modules, each processing one type of dynamic feature. The module parameters of multiple dynamic feature processing modules can be the same or different, but their structures are usually similar.

[0058] For each dynamic feature processing module, the dynamic features corresponding to the dynamic feature processing module and the static facial features of the target object are input into the dynamic feature processing module; the static facial features and dynamic features are fused through the attention network in the dynamic feature extraction module to obtain fused features; wherein, the fused features include: common key features representing emotions between static facial features and dynamic features.

[0059] For example, if the dynamic feature processing model corresponds to facial expression change features, then only the facial expression change features and static facial features of the target object are input, without inputting other dynamic features. The dynamic feature processing model includes an attention network. Since the attention network has the function of strengthening local features, it can identify common key features representing emotions in static facial features and dynamic features. Then, during the feature fusion process, it strengthens the common key features, and emotion analysis based on these features can improve the accuracy of emotion analysis.

[0060] In actual implementation, dynamic features need to be preprocessed before being input into the dynamic feature processing model. Specifically, the dynamic features are input into the time series model, and the context features of the dynamic features are output. The context features are used to characterize the changing trend of the features over time. Temporal position information is added to the context features, and the context features with added temporal position information are determined as dynamic features to be input into the dynamic feature extraction module. The temporal position information is used to indicate the sequential position of the features in the context features over time.

[0061] The aforementioned temporal model can be implemented using various models such as GRU (Gated Recurrent Unit), RNN (Recurrent Neural Network), and LSTM (Long Short-Term Memory). After processing by the temporal model, dynamic features are output as contextual features. These contextual features represent features through time parameters. Dynamic features typically contain a series of features; after processing by the temporal model, the temporal relationships between these features can be obtained. Therefore, contextual features are used to represent the changing trends of features in dynamic features over time. Furthermore, temporal position information is added to the contextual features, further indicating the sequential temporal relationships between the various features within the contextual features.

[0062] To further improve the effect of feature fusion, in one specific implementation, the attention network in the dynamic feature processing module includes a self-attention network and a cross-attention network; wherein, the self-attention network can be a multi-head attention network structure, and the cross-attention network can be a network result of a multi-head attention network.

[0063] First, the dynamic features are transformed to obtain the first input parameters of the self-attention network. The first input parameters are then input into the self-attention network to output intermediate features of the dynamic features. The intermediate features are used to characterize the autocorrelation of the dynamic features. The second input parameters of the cross-attention network are determined based on the intermediate features, and the third input parameters of the cross-attention network are determined based on the static facial features. The second and third input parameters are then input into the cross-attention network to obtain the output results. The fusion features are then determined based on the output results.

[0064] Specifically, attention networks typically require three input parameters: K, V, and Q. These values ​​are obtained by transforming dynamic features. The values ​​of K, V, and Q can be the same or different. When the arrays of K, V, and Q values ​​are different, different transformation methods can be applied to the dynamic features. For example, a first transformation method can be used to transform the dynamic features and obtain the K value. This first transformation method can be a convolutional transformation, a mapping transformation, or similar method, and the parameters used for this transformation method can be preset. A second transformation method can be used to transform the dynamic features and obtain the V value. This second transformation method can also be a convolutional transformation, a mapping transformation, or similar method, and the parameters used for this transformation method can be preset. A third transformation method can be used to transform the dynamic features and obtain the Q value. This third transformation method can also be a convolutional transformation, a mapping transformation, or similar method, and the parameters used for this transformation method can be preset. The aforementioned first, second, and third transformation methods can use the same transformation method or different transformation methods. When the transformation methods are the same, the same transformation parameters or different transformation parameters can be used. It should be noted that the three input parameters in the first input parameter of the self-attention network are all obtained from the changes in dynamic features. Therefore, the three input parameters can be understood as expressions of different levels or different dimensions of dynamic features.

[0065] Self-attention networks are primarily used to calculate global self-attention results for a dynamic feature. The first input parameters are all obtained by transforming the dynamic feature. The self-attention network processes the first input parameters and outputs intermediate features, which represent the autocorrelation of the dynamic feature. In one approach, the first input parameters include K, V, and Q values; where K, V, and Q are all in matrix form. The self-attention network calculates the K, V, and Q values ​​of the first input parameters as follows:

[0066] A=K T Q;

[0067] A' = softmax(A);

[0068] O=VA';

[0069] Among them, K T is the transpose of K; softmax represents the normalized exponential function; O represents the intermediate features of the dynamic features output by the self-attention network.

[0070] Unlike self-attention networks, some parameters of cross-attention networks are determined based on intermediate features output by self-attention networks, meaning these parameters are related to dynamic features. Furthermore, another part of the parameters of cross-attention networks are determined based on static facial features, meaning these parameters are related to static facial features. Since the input parameters are related to both dynamic and static facial features, cross-attention networks can achieve the fusion of dynamic and static facial features, thereby obtaining fused features.

[0071] In one specific implementation, intermediate features and dynamic features undergo a first fusion process to obtain a first fusion result. This first fusion result is then transformed to obtain the second input parameters of the cross-attention network. Specifically, the first fusion process can involve adding the intermediate and dynamic features to obtain a sum, and then normalizing this sum to obtain the final fusion result. The feature addition process can specifically involve feature concatenation or adding feature data at the same location. When transforming the first fusion result, the transformation method can be convolutional transformation, mapping transformation, or other transformation methods, and the parameters used for the transformation method can be preset.

[0072] In addition, static facial features are transformed to obtain the third input parameters for the cross-attention network. The transformation process can include convolutional transformations, mapping transformations, etc., and the parameters used for each transformation can be preset.

[0073] As an example, the second input parameter mentioned above includes the K and V values ​​of the cross-attention network, and the third input parameter includes the Q value of the cross-attention network; the values ​​of K and V can be the same or different. The cross-attention network is used to calculate the cross-attention result of static facial features and dynamic features; in one approach, the attention coefficient of the cross-attention network is calculated as follows:

[0074]

[0075]

[0076] in, The attention coefficient represents the cross-attention network; These are intermediate features output by the aforementioned self-attention network; Static facial features; for The transpose of the matrix; for Dimensions.

[0077] Furthermore, the output of the cross-attention network needs to be processed as follows to obtain the fusion features: the first fusion result corresponding to the output result and the second input parameter is subjected to a second fusion process to obtain a second fusion result; wherein the second input parameter is obtained by transforming the first fusion result; the second fusion result is input into a preset first multilayer perceptron, and the second fusion result is mapped by the first multilayer perceptron to obtain a mapping result; the mapping result and the second fusion result are subjected to a third fusion process to obtain the fusion features.

[0078] The first fusion result mentioned above is specifically the first fusion result obtained after performing a first fusion process on the intermediate features and dynamic features output by the self-attention network. The second fusion process mentioned above can specifically involve adding features to the output result and the first fusion result, obtaining a summed result, and then normalizing the summed result to obtain the second fusion result. The feature addition process can specifically be feature concatenation or adding feature data at the same position. The third fusion process mentioned above can specifically involve adding features to the mapping result and the second fusion result, obtaining a summed result, and then normalizing the summed result to obtain fused features. The feature addition process can specifically be feature concatenation or adding feature data at the same position. The first multilayer perceptron mentioned above can be implemented using an MLP (Multilayer Perceptron) network.

[0079] Figure 2 This illustrates a module structure and data flow for the dynamic feature processing module. In this example, the self-attention network is implemented using a multi-head self-attention network, and the cross-attention network is implemented using a multi-head cross-attention network. The first fusion processing network performs the first fusion processing described in the previous embodiment, the second fusion processing network performs the second fusion processing described in the previous embodiment, and the third fusion processing network performs the third fusion processing described in the previous embodiment. All three fusion processing networks can be implemented using Add&Norm networks to perform addition and normalization of the input features.

[0080] When multiple dynamic features are involved, each dynamic feature can be processed by a corresponding dynamic feature processing module to obtain a fused feature. In the aforementioned object sentiment analysis model, multiple dynamic feature processing modules are included. These modules take into account the corresponding type of dynamic feature and output the fused feature corresponding to that feature. The fused feature output by the dynamic feature processing module is then concatenated with static facial features to obtain a concatenated feature. This concatenated feature is input into a pre-defined second multilayer perceptron to output the sentiment analysis result of the target object. The concatenated feature includes fused features and static facial features. The dimensionality of a specified dimension of the concatenated feature is typically equal to the sum of the dimensionalities of the specified dimensions of each fused feature and static facial feature. The second multilayer perceptron can be implemented using an MLP network.

[0081] Furthermore, to make the sentiment analysis results more accurate and reasonable, in this embodiment, the object sentiment analysis model outputs analysis results from multiple sentiment analysis methods. Based on this, the aforementioned second multilayer perceptron includes multiple branch networks; during training, each branch network learns a feature mapping method corresponding to a sentiment analysis method. The concatenated features are input into the multiple branch networks of the second multilayer perceptron; wherein, each branch network has a preset feature mapping method; this feature mapping method includes several of the following: linear combination mapping based on preset facial action units, linear combination mapping based on preset multiple basic emotion types, and linear representation mapping based on the positive / negative degree and intensity of emotions; the concatenated features are mapped according to the feature mapping method corresponding to the branch network to obtain the sentiment analysis results output by the branch network.

[0082] In the feature mapping method based on linear combination mapping of pre-defined facial motion units, the face is pre-divided into multiple motion units according to the distribution of facial muscles. When the face expresses emotions through facial expressions, the expressions are represented by linear combinations of motion units. After receiving the spliced ​​features, the branch network specifically calculates the linear weight of each motion unit according to the spliced ​​features, and then performs linear combination of the motion units using these linear weights to obtain the emotion analysis results.

[0083] In a feature mapping method based on linear combination mapping of multiple pre-defined basic emotion types, emotions are pre-classified into multiple basic emotions such as neutral, happy, sad, surprised, afraid, angry, and disgusted. After receiving the spliced ​​features, the branch network specifically calculates the linear weight of each basic emotion according to the spliced ​​features, and then linearly combines the basic emotions using these linear weights to obtain the emotion analysis results.

[0084] In the feature mapping method that uses linear representation mapping based on the positive / negative degree and intensity of emotions, after the branch network receives the spliced ​​features, its feature mapping method is as follows: calculate the parameters of positive / negative degree and intensity according to the spliced ​​features, and represent emotions based on these two parameters to obtain the emotion analysis results.

[0085] In actual implementation, the aforementioned second multilayer perceptron includes three branch networks, which correspond to three feature mapping methods: linear combination mapping based on preset facial action units, linear combination mapping based on preset multiple basic emotion types, and linear representation mapping based on the positive and negative degree and intensity of emotions. Therefore, the obtained emotion analysis results include the emotion analysis results obtained by each feature mapping method.

[0086] Figure 3 This example illustrates a modular structure and data flow of an object sentiment analysis model. The model includes multiple dynamic feature processing modules and a second multilayer perceptron. Voice features are processed by temporal model 1 and then input to dynamic feature processing module 1 to obtain fused features corresponding to the voice features. Facial expression change features are processed by temporal model 2 and then input to dynamic feature processing module 2 to obtain fused features corresponding to the facial expression change features. Language content features are processed by temporal model 3 and then input to dynamic feature processing module 3 to obtain fused features corresponding to the language content features. These three fused features are concatenated with static facial features and then input to the second multilayer perceptron. Branch network 1 uses a feature mapping method based on linear combination mapping of facial action units to obtain sentiment analysis results based on action units. Branch network 2 uses a feature mapping method based on linear combination mapping of multiple basic emotion types to obtain sentiment analysis results based on basic emotion types. Branch network 3 uses a feature mapping method based on linear representation mapping of the positive / negative degree and intensity of emotions to obtain sentiment analysis results based on arousal-stimulation.

[0087] In one specific implementation, the object emotion analysis method described in the foregoing embodiments can be used to analyze the emotion of a target object in each video frame in video data, thereby obtaining the emotion changes of the target object in the video data.

[0088] Specifically, the aforementioned multimedia data refers to video data containing the target object within the video frame. Each video frame in the video data is taken as the current video frame, and the following operations are performed on the current video frame: static facial features are obtained from the current video frame; facial expression change features are obtained from the video frame sequence containing the current video frame; sound features are obtained from the audio data corresponding to the video frame sequence; and language content features are obtained from the audio data and / or subtitle data corresponding to the video frame sequence. Through the object emotion analysis method described in the above embodiments, the emotion analysis result of the target object in the current video frame is obtained.

[0089] For example, the aforementioned video data includes N video frames. For video frame i, a sentiment analysis result i can be obtained. Arranging the sentiment analysis results of each video frame according to the order of the video frames yields sentiment analysis result 1, sentiment analysis result 2, ..., sentiment analysis result N. In one example, the sentiment analysis result corresponding to the video data can specifically be a series of emotional combinations such as peaceful, peaceful, surprised, surprised, surprised, happy, happy, happy, happy, happy, etc.

[0090] In addition, to further improve the accuracy of sentiment analysis results, after obtaining the sentiment analysis results of the target object in the current video frame, the sentiment analysis results of the corresponding video frames in the video data are obtained; based on the sentiment analysis results of the corresponding video frames, it is determined that there are video frames with sudden mood changes in the video data; among them, the sentiment analysis results of the video frames with sudden mood changes are the first analysis results, and the sentiment analysis results of the video frames before and after the video frames with sudden mood changes are the second analysis results; the first analysis results are different from the second analysis results; based on the second analysis results, the sentiment analysis results of the video frames with sudden mood changes are smoothed to obtain the smoothed sentiment analysis results.

[0091] It is understandable that emotional changes in humans or other subjects are usually smooth, with rare sudden mood swings. The aforementioned video frame with a sudden mood swing can be understood as one whose sentiment analysis result differs from both the sentiment analysis results of the preceding and following videos. In one example, the sentiment analysis results for the video data are: peaceful, peaceful, surprised, surprised, surprised, happy, angry, happy, happy. The video frame corresponding to "anger" is the aforementioned video frame with a sudden mood swing. Since the target subject typically does not suddenly become angry while in a happy mood, this indicates that the sentiment analysis result for this video frame with a sudden mood swing may be incorrect. To avoid this error, the sentiment analysis result of the video frame with a sudden mood swing is smoothed based on the sentiment analysis results of the preceding and following video frames. Continuing with the example above, the sentiment analysis result for the smoothed video data is: peaceful, peaceful, surprised, surprised, surprised, happy, happy, happy, happy.

[0092] This method can avoid obvious errors in sentiment analysis results, thereby improving the accuracy and reasonableness of the results.

[0093] The following examples describe the training method of the aforementioned object sentiment analysis model.

[0094] Taking human emotion analysis as an example, this model can be trained on a dataset of facial expressions in natural environments, which contains a large number of spontaneous emotion expression videos from real, non-experimental environments. Since the object emotion analysis model needs to output multiple emotion analysis results, multiple loss functions are required to constrain the model during training.

[0095] Specifically, for sentiment analysis results based on action units, the following loss function is used to calculate the loss value. :

[0096] =

[0097]

[0098]

[0099]

[0100]

[0101] in, This represents the number of action units (AUs) to be predicted. The label represents the j-th AU, and its value is 0 or 1, indicating whether the AU appears. This represents the network's predicted probability for the j-th AU. This represents the direct output of the aforementioned branch network 1; It is the direct output of the network. After a sigmoid algorithm is applied, if the model needs to predict a total of N action units, it will output N labels. For a given label, a value of 1 indicates that the corresponding action unit has appeared, while a value of 0 indicates that the corresponding action unit has not appeared. The model will output the predicted probability for each action unit. There are two sets, where, This represents the set of AU indices with a label value of 0 out of N AUs. Let i represent the set of AU indices with a label value of 1 out of N AUs, where i and j are the AU indices. i represents... AU, "If" means "if".

[0102] It is the cross-entropy loss function. This ensures that the output of all appearing AUs is greater than 0, and the output of all non-appearing AUs is less than 0, which helps to characterize the symbiotic relationship between AUs.

[0103] For sentiment analysis results based on multiple basic sentiment types, the cross-entropy loss function with soft labels is used to calculate the loss value. :

[0104]

[0105] in, It is the number of basic emotion types. It is the predicted probability of the i-th emotion type, and e is the actual emotion type; The predicted probability of the e-th emotion type A value of 0.9 indicates the degree of relaxation of the soft label.

[0106] For sentiment analysis results based on the positive / negative degree and intensity of emotions, the loss value is calculated using the Concordance Correlation Coefficient (CCC) loss function.

[0107]

[0108] Here, V represents a label indicating the degree of positive or negative. This indicates the degree of positivity or negativity of the actual output of branch network 3; A represents the label indicating the intensity. This indicates the intensity of the actual output of branch network 3.

[0109] The final loss function of the object sentiment analysis model is:

[0110]

[0111] in, The value can be 0 or 1, and is used to determine whether the training sample has a label for facial action units; The value is 0 or 1, and is used to determine whether the training samples have labels for basic emotion types. The value can be 0 or 1, and is used to indicate whether the training sample has a label indicating the positive or negative degree and intensity of the emotion. During training, some training samples may only have partial labels. In this case, other known labels can be used to compensate for the lack of relevant information for the unknown labels.

[0112] Under the supervision of multiple loss functions, an optimizer is used to train and optimize the network. When the loss value L converges, the model training ends, and the network parameters in the model can be fixed. This yields an object sentiment analysis model that can accurately analyze emotions and has good generalization ability, and the sentiment analysis results can be represented in multiple ways.

[0113] This embodiment provides a method for analyzing object emotions. Considering that humans perceive emotions primarily through multiple modalities such as vision, hearing, and text, tasks related to facial expressions are closely linked to this modality. This embodiment proposes utilizing multimodal dynamic features to fully mine emotion-related information, assisting in facial expression detection tasks. Furthermore, since the extracted multimodal dynamic features are independent of personalized or identity information, they can effectively help alleviate identity overfitting. Further, to better integrate the multimodal emotion features, a multi-head attention mechanism is proposed for feature fusion, making the extracted features more conducive to improving the performance of downstream tasks. Additionally, considering that there are very few abrupt changes in basic emotion labels within a continuous frame, this embodiment also proposes using a filtering strategy to further smooth the prediction results, removing some erroneous predictions and further improving the accuracy and rationality of emotion analysis.

[0114] For the corresponding method embodiments described above, see [link to relevant documentation]. Figure 4 The diagram shows a structural schematic of an object emotion analysis device, which includes:

[0115] The feature extraction module 40 is used to acquire multimedia data associated with the target object and extract static facial features and dynamic features of the target object from the multimedia data; wherein, the dynamic features include one or more of the target object's facial expression change features, voice features and language content features;

[0116] The result output module 42 is used to input static facial features and dynamic features into a pre-trained object sentiment analysis model, perform feature fusion processing on the static facial features and dynamic features through the object sentiment analysis model to obtain fused features, and output the sentiment analysis results of the target object based on the fused features.

[0117] The aforementioned object emotion analysis device acquires multimedia data associated with the target object, extracts static facial features and dynamic features of the target object from the multimedia data; wherein, dynamic features include one or more of the target object's facial expression changes, vocal features, and language content features; the static facial features and dynamic features are input into a pre-trained object emotion analysis model, and the object emotion analysis model performs feature fusion processing on the static facial features and dynamic features to obtain fused features; based on the fused features, the object emotion analysis result is output. In this method, the object emotion analysis model performs feature fusion processing on the static facial features and dynamic features of the target object to output the object's emotion analysis result. Since dynamic features also contain feature information representing emotions, combining static facial features with dynamic features for emotion analysis can, to a certain extent, reduce the influence of interfering features in static facial features on the emotion analysis result, strengthen the role of feature information representing emotions, and thus improve the accuracy of the emotion analysis result.

[0118] The aforementioned feature extraction module is further configured to: if the dynamic features include facial expression change features, obtain a specified video frame sequence from the video data in the multimedia data; wherein, in the video frame sequence, the facial expression of the target object changes dynamically; and extract the facial expression change features of the target object from the video frame sequence using a pre-trained facial expression feature extraction model.

[0119] The aforementioned feature extraction module is further configured to: if the dynamic features include sound features, obtain a specified audio sequence from the audio data in the multimedia data; wherein the audio sequence includes sound signals emitted by the target object; and extract the sound features of the target object from the audio sequence using a pre-trained sound feature extraction model; wherein the sound features include one or more of frequency features, pitch features, volume features, and energy features.

[0120] The aforementioned feature extraction module is further configured to: if the dynamic features include language content features, obtain the language content text of the target object from the subtitle data and / or audio data in the multimedia data; extract the language content features of the target object from the language content text using a pre-trained language content feature extraction model; wherein, the language content features are used to: characterize the language meaning of the language spoken by the target object.

[0121] The aforementioned object emotion analysis model includes at least one dynamic feature processing module, which is used to input dynamic features of the corresponding type. The aforementioned result output module is also used to: input the dynamic features corresponding to the dynamic feature processing module and the static facial features of the target object into the dynamic feature processing module; perform feature fusion processing on the static facial features and dynamic features through the attention network in the dynamic feature extraction module to obtain fused features; wherein, the fused features include: common key features representing emotions between the static facial features and dynamic features.

[0122] The aforementioned device further includes: a time-series processing module, used to: input dynamic features into a time-series model and output context features of the dynamic features; wherein, the context features are used to: characterize the changing trend of features in the dynamic features over time; add time-series position information to the context features, and determine the context features with added time-series position information as dynamic features to be input into the dynamic feature extraction module; wherein, the time-series position information is used to: indicate the sequential position of features in the context features over time.

[0123] The attention network in the dynamic feature processing module includes a self-attention network and a cross-attention network. The output module is further configured to: transform the dynamic features to obtain the first input parameters of the self-attention network, input the first input parameters into the self-attention network, and output intermediate features of the dynamic features; wherein the intermediate features are used to: characterize the autocorrelation of the dynamic features; determine the second input parameters of the cross-attention network based on the intermediate features, determine the third input parameters of the cross-attention network based on the static facial features, input the second and third input parameters into the cross-attention network to obtain the output results, and determine the fusion features based on the output results.

[0124] The above-mentioned output module is also used to: perform a first fusion process on the intermediate features and dynamic features to obtain a first fusion result; and perform a transformation process on the first fusion result to obtain the second input parameters of the cross-attention network.

[0125] The above output module is also used to: transform static facial features to obtain the third input parameter of the cross-attention network.

[0126] The above-mentioned output module is further configured to: perform a second fusion process on the output result and the first fusion result corresponding to the second input parameter to obtain a second fusion result; wherein the second input parameter is obtained by transforming the first fusion result; input the second fusion result into a preset first multilayer perceptron, perform mapping processing on the second fusion result through the first multilayer perceptron to obtain a mapping result; and perform a third fusion process on the mapping result and the second fusion result to obtain a fusion feature.

[0127] The aforementioned object sentiment analysis model includes multiple dynamic feature processing modules. The dynamic feature processing modules are used to input dynamic features of the corresponding type and output fused features corresponding to the dynamic features. The aforementioned result output module is also used to: splice the fused features output by the dynamic feature processing modules and the static facial features to obtain spliced ​​features; input the spliced ​​features into a preset second multilayer perceptron and output the sentiment analysis results of the target object.

[0128] The aforementioned second multilayer perceptron includes multiple branch networks; the aforementioned result output module is further used to: input the spliced ​​features into the multiple branch networks of the second multilayer perceptron respectively; wherein, each branch network has a preset feature mapping method corresponding to the branch network; the feature mapping method includes multiple of the following: linear combination mapping based on preset facial action units, linear combination mapping based on preset multiple basic emotion types, and linear representation mapping based on the positive / negative degree and intensity of emotions; the spliced ​​features are mapped by the branch networks according to the feature mapping method corresponding to the branch networks to obtain the emotion analysis results output by the branch networks.

[0129] The aforementioned multimedia data refers to video data containing the target object within the video frame; the aforementioned feature extraction module is further configured to: sequentially take each video frame in the video data as the current video frame, and perform the following operations on the current video frame: obtain static facial features from the current video frame; obtain facial expression change features from the video frame sequence containing the current video frame; obtain sound features from the audio data corresponding to the video frame sequence; and obtain language content features from the audio data and / or subtitle data corresponding to the video frame sequence.

[0130] The aforementioned device further includes a smoothing module, configured to: acquire the sentiment analysis results corresponding to video frames in the video data; determine, based on the sentiment analysis results corresponding to the video frames, that there are video frames with sudden mood changes in the video data; wherein, the sentiment analysis result of the video frame with sudden mood changes is the first analysis result, and the sentiment analysis results of the adjacent video frames before and after the video frame with sudden mood changes are all the second analysis results; the first analysis result is different from the second analysis result; and based on the second analysis result, smooth the sentiment analysis results of the video frame with sudden mood changes to obtain the smoothed sentiment analysis result.

[0131] This embodiment also provides an electronic device, including a processor and a memory. The memory stores machine-executable instructions that can be executed by the processor. The processor executes the machine-executable instructions to implement the aforementioned method for analyzing object emotions. This electronic device can be implemented as a server, or as any combination of two or more of various types of user terminals such as laptops, tablets, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or these data processing devices.

[0132] See Figure 5 As shown, the electronic device includes a processor 100 and a memory 101. The memory 101 stores machine-executable instructions that can be executed by the processor 100. The processor 100 executes the machine-executable instructions to implement the above-mentioned method for analyzing object emotions.

[0133] Furthermore, Figure 5 The electronic device shown also includes a bus 102 and a communication interface 103, with the processor 100, the communication interface 103 and the memory 101 connected via the bus 102.

[0134] The memory 101 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 103 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 102 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0135] Processor 100 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 100 or by instructions in software form. Processor 100 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a readily available storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 101, and the processor 100 reads the information from memory 101 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0136] The processor in the aforementioned electronic device, by executing machine-executable instructions, can perform the following operations in the aforementioned object emotion analysis method: acquiring multimedia data associated with the target object; extracting static facial features and dynamic features of the target object from the multimedia data; wherein, the dynamic features include one or more of the target object's facial expression change features, voice features, and language content features; inputting the static facial features and dynamic features into a pre-trained object emotion analysis model; performing feature fusion processing on the static facial features and dynamic features through the object emotion analysis model to obtain fused features; and outputting the object's emotion analysis result based on the fused features.

[0137] In this approach, the static and dynamic facial features of the target object are fused using an object sentiment analysis model to output the sentiment analysis results. Since dynamic features also contain information representing emotions, combining static facial features with dynamic features for sentiment analysis can, to some extent, reduce the influence of interfering features in static facial features on the sentiment analysis results, strengthen the role of information representing emotions, and thus improve the accuracy of the sentiment analysis results.

[0138] The processor in the aforementioned electronic device can execute machine-executable instructions to perform the following operations in the aforementioned method for analyzing the emotion of an object: if the dynamic features include facial expression change features, obtain a specified video frame sequence from the video data in the multimedia data; wherein, in the video frame sequence, the facial expression of the target object changes dynamically; and extract the facial expression change features of the target object from the video frame sequence using a pre-trained facial expression feature extraction model.

[0139] The processor in the aforementioned electronic device, by executing machine-executable instructions, can perform the following operations in the aforementioned method for analyzing the emotion of an object: if the dynamic features include sound features, obtain a specified audio sequence from the audio data in the multimedia data; wherein the audio sequence includes a sound signal emitted by the target object; extract the sound features of the target object from the audio sequence using a pre-trained sound feature extraction model; wherein the sound features include one or more of frequency features, pitch features, volume features, and energy features.

[0140] The processor in the aforementioned electronic device, by executing machine-executable instructions, can perform the following operations in the aforementioned method for analyzing the emotion of an object: if the dynamic features include language content features, obtain the language content text of the target object from the subtitle data and / or audio data in the multimedia data; extract the language content features of the target object from the language content text using a pre-trained language content feature extraction model; wherein, the language content features are used to: characterize the linguistic meaning of the language spoken by the target object.

[0141] In the above approach, adding multimodal dynamic features can alleviate the overfitting problem, prevent the model from learning personalized information, focus on learning emotion-related features, and improve the accuracy of the model in analyzing emotions.

[0142] The processor in the aforementioned electronic device, by executing machine-executable instructions, can implement the following operations in the aforementioned object emotion analysis method: the object emotion analysis model includes at least one dynamic feature processing module, which is used to input dynamic features of the corresponding type; for the dynamic feature processing module, the dynamic features corresponding to the dynamic feature processing module and the static facial features of the target object are input to the dynamic feature processing module; the static facial features and dynamic features are fused through the attention network in the dynamic feature extraction module to obtain fused features; wherein, the fused features include: common key features representing emotions between static facial features and dynamic features.

[0143] The processor in the aforementioned electronic device, by executing machine-executable instructions, can perform the following operations in the aforementioned method for analyzing object emotions: inputting dynamic features into a temporal model and outputting contextual features of the dynamic features; wherein, the contextual features are used to: characterize the changing trend of features in the dynamic features over time; adding temporal position information to the contextual features, and determining the contextual features with added temporal position information as dynamic features to be input into the dynamic feature extraction module; wherein, the temporal position information is used to: indicate the sequential position of features in the contextual features over time.

[0144] The processor in the aforementioned electronic device, by executing machine-executable instructions, can implement the following operations in the aforementioned method for analyzing object emotions: the attention network in the dynamic feature processing module includes a self-attention network and a cross-attention network; the dynamic features are transformed to obtain the first input parameters of the self-attention network, the first input parameters are input into the self-attention network, and intermediate features of the dynamic features are output; wherein, the intermediate features are used to: characterize the autocorrelation of the dynamic features; the second input parameters of the cross-attention network are determined based on the intermediate features, the third input parameters of the cross-attention network are determined based on the static facial features, the second input parameters and the third input parameters are input into the cross-attention network to obtain the output result, and the fusion feature is determined based on the output result.

[0145] The processor in the aforementioned electronic device can perform the following operations in the above-mentioned object emotion analysis method by executing machine-executable instructions: performing a first fusion process on intermediate features and dynamic features to obtain a first fusion result; and performing a transformation process on the first fusion result to obtain the second input parameters of the cross-attention network.

[0146] The processor in the aforementioned electronic device can perform the following operations in the above-mentioned method for analyzing the emotion of an object by executing machine-executable instructions: transforming static facial features to obtain the third input parameter of the cross-attention network.

[0147] The processor in the aforementioned electronic device, by executing machine-executable instructions, can perform the following operations in the aforementioned method for analyzing object emotions: performing a second fusion process on the first fusion result corresponding to the output result and the second input parameter to obtain a second fusion result; wherein the second input parameter is obtained after transformation processing of the first fusion result; inputting the second fusion result into a preset first multilayer perceptron, and performing mapping processing on the second fusion result through the first multilayer perceptron to obtain a mapping result; performing a third fusion process on the mapping result and the second fusion result to obtain a fusion feature.

[0148] The processor in the aforementioned electronic device can execute machine-executable instructions to perform the following operations in the above-mentioned object emotion analysis method: the object emotion analysis model includes multiple dynamic feature processing modules, which are used to input dynamic features of corresponding types and output fused features corresponding to the dynamic features; the fused features output by the dynamic feature processing modules and static facial features are spliced ​​together to obtain spliced ​​features; the spliced ​​features are input into a preset second multilayer perceptron and the emotion analysis results of the target object are output.

[0149] The processor in the aforementioned electronic device, by executing machine-executable instructions, can perform the following operations in the aforementioned method for analyzing the emotion of an object: The second multilayer perceptron includes multiple branch networks; the spliced ​​features are input into the multiple branch networks of the second multilayer perceptron; wherein, each branch network has a preset feature mapping method; the feature mapping method includes several of the following: linear combination mapping based on preset facial action units, linear combination mapping based on preset multiple basic emotion types, and linear representation mapping based on the positive / negative degree and intensity of the emotion; the spliced ​​features are mapped according to the feature mapping method corresponding to the branch network through the branch networks to obtain the emotion analysis results output by the branch networks.

[0150] In the above methods, the object sentiment analysis model outputs analysis results from multiple sentiment analysis methods, which can make the sentiment analysis results more accurate and reasonable.

[0151] The processor in the aforementioned electronic device, by executing machine-executable instructions, can implement the following operations in the aforementioned method for analyzing the object's emotions: the multimedia data is video data containing the target object in a video frame; each video frame in the video data is taken as the current video frame, and the following operations are performed on the current video frame: static facial features are obtained from the current video frame; facial expression change features are obtained from the video frame sequence containing the current video frame; sound features are obtained from the audio data corresponding to the video frame sequence; and language content features are obtained from the audio data and / or subtitle data corresponding to the video frame sequence.

[0152] The processor in the aforementioned electronic device, by executing machine-executable instructions, can perform the following operations in the aforementioned method for analyzing the emotion of an object: acquiring the emotion analysis results corresponding to video frames in video data; determining, based on the emotion analysis results corresponding to the video frames, that there are video frames in the video data with sudden emotional changes; wherein, the emotion analysis results of the video frames with sudden emotional changes are the first analysis results, and the emotion analysis results of the video frames before and after the video frames with sudden emotional changes are the second analysis results; the first analysis results are different from the second analysis results; and based on the second analysis results, smoothing is performed on the emotion analysis results of the video frames with sudden emotional changes to obtain smoothed emotion analysis results.

[0153] This method can avoid obvious errors in sentiment analysis results, thereby improving the accuracy and reasonableness of the results.

[0154] This embodiment also provides a machine-readable storage medium storing machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to implement the above-mentioned method for analyzing object emotions.

[0155] The machine-executable instructions stored in the aforementioned machine-readable storage medium can be executed to perform the following operations in the aforementioned object emotion analysis method: acquiring multimedia data associated with the target object; extracting static facial features and dynamic features of the target object from the multimedia data; wherein, the dynamic features include one or more of the target object's facial expression change features, voice features, and language content features; inputting the static facial features and dynamic features into a pre-trained object emotion analysis model; performing feature fusion processing on the static facial features and dynamic features through the object emotion analysis model to obtain fused features; and outputting the object's emotion analysis result based on the fused features.

[0156] In this approach, the static and dynamic facial features of the target object are fused using an object sentiment analysis model to output the sentiment analysis results. Since dynamic features also contain information representing emotions, combining static facial features with dynamic features for sentiment analysis can, to some extent, reduce the influence of interfering features in static facial features on the sentiment analysis results, strengthen the role of information representing emotions, and thus improve the accuracy of the sentiment analysis results.

[0157] The machine-executable instructions stored in the aforementioned machine-readable storage medium can be executed to perform the following operations in the above-mentioned method for analyzing the emotion of an object: if the dynamic features include facial expression change features, a specified video frame sequence is obtained from the video data in the multimedia data; wherein, the facial expression of the target object changes dynamically in the video frame sequence; and the facial expression change features of the target object are extracted from the video frame sequence using a pre-trained facial expression feature extraction model.

[0158] The machine-executable instructions stored in the aforementioned machine-readable storage medium can be executed to perform the following operations in the aforementioned method for analyzing the emotion of an object: if the dynamic features include sound features, obtain a specified audio sequence from the audio data in the multimedia data; wherein the audio sequence includes a sound signal emitted by the target object; extract the sound features of the target object from the audio sequence using a pre-trained sound feature extraction model; wherein the sound features include one or more of frequency features, pitch features, volume features, and energy features.

[0159] The machine-executable instructions stored in the aforementioned machine-readable storage medium can be executed to perform the following operations in the aforementioned method for analyzing the emotion of an object: if the dynamic features include language content features, obtain the language content text of the target object from the subtitle data and / or audio data in the multimedia data; extract the language content features of the target object from the language content text using a pre-trained language content feature extraction model; wherein, the language content features are used to: characterize the linguistic meaning of the language spoken by the target object.

[0160] In the above approach, adding multimodal dynamic features can alleviate the overfitting problem, prevent the model from learning personalized information, focus on learning emotion-related features, and improve the accuracy of the model in analyzing emotions.

[0161] The machine-executable instructions stored in the aforementioned machine-readable storage medium can be used to perform the following operations in the object emotion analysis method: the object emotion analysis model includes at least one dynamic feature processing module, which is used to input dynamic features of the corresponding type; for the dynamic feature processing module, the dynamic features corresponding to the dynamic feature processing module and the static facial features of the target object are input to the dynamic feature processing module; the static facial features and dynamic features are fused through the attention network in the dynamic feature extraction module to obtain fused features; wherein, the fused features include: common key features representing emotions between static facial features and dynamic features.

[0162] The machine-executable instructions stored in the aforementioned machine-readable storage medium can be used to perform the following operations in the above-mentioned object sentiment analysis method by executing the machine-executable instructions: inputting dynamic features into a time-series model and outputting context features of the dynamic features; wherein, the context features are used to: characterize the trend of feature changes over time in the dynamic features; adding time-series position information to the context features, and determining the context features with added time-series position information as dynamic features to be input into the dynamic feature extraction module; wherein, the time-series position information is used to: indicate the sequential position of features in the context features over time.

[0163] The machine-executable instructions stored in the aforementioned machine-readable storage medium can be executed to perform the following operations in the above-mentioned method for analyzing object emotions: the attention network in the dynamic feature processing module includes a self-attention network and a cross-attention network; the dynamic features are transformed to obtain the first input parameters of the self-attention network, the first input parameters are input into the self-attention network, and intermediate features of the dynamic features are output; wherein, the intermediate features are used to: characterize the autocorrelation of the dynamic features; the second input parameters of the cross-attention network are determined based on the intermediate features, the third input parameters of the cross-attention network are determined based on the static facial features, the second input parameters and the third input parameters are input into the cross-attention network to obtain the output result, and the fusion feature is determined based on the output result.

[0164] The machine-executable instructions stored in the aforementioned machine-readable storage medium can be used to perform the following operations in the above-mentioned object emotion analysis method by executing the machine-executable instructions: performing a first fusion process on intermediate features and dynamic features to obtain a first fusion result; and transforming the first fusion result to obtain the second input parameters of the cross-attention network.

[0165] The machine-executable instructions stored in the aforementioned machine-readable storage medium can be used to perform the following operations in the above-mentioned method for analyzing object emotions by executing the machine-executable instructions: transforming static facial features to obtain the third input parameter of the cross-attention network.

[0166] The machine-executable instructions stored in the aforementioned machine-readable storage medium can be executed to perform the following operations in the aforementioned method for analyzing object emotions: performing a second fusion process on the first fusion result corresponding to the output result and the second input parameter to obtain a second fusion result; wherein the second input parameter is obtained by transforming the first fusion result; inputting the second fusion result into a preset first multilayer perceptron, and performing a mapping process on the second fusion result through the first multilayer perceptron to obtain a mapping result; and performing a third fusion process on the mapping result and the second fusion result to obtain a fusion feature.

[0167] The machine-executable instructions stored in the aforementioned machine-readable storage medium can be used to execute the following operations in the above-mentioned object emotion analysis method: The object emotion analysis model includes multiple dynamic feature processing modules. The dynamic feature processing modules are used to input dynamic features of corresponding types and output fused features corresponding to the dynamic features; the fused features output by the dynamic feature processing modules and static facial features are spliced ​​together to obtain spliced ​​features; the spliced ​​features are input into a preset second multilayer perceptron and the emotion analysis results of the target object are output.

[0168] The machine-executable instructions stored in the aforementioned machine-readable storage medium can be executed to perform the following operations in the above-mentioned method for analyzing object emotions: The second multilayer perceptron includes multiple branch networks; the spliced ​​features are respectively input into the multiple branch networks of the second multilayer perceptron; wherein, each branch network has a preset feature mapping method; the feature mapping method includes several of the following: linear combination mapping based on preset facial action units, linear combination mapping based on preset multiple basic emotion types, and linear representation mapping based on the positive / negative degree and intensity of emotions; the spliced ​​features are mapped according to the feature mapping method corresponding to the branch network to obtain the emotion analysis result output by the branch network.

[0169] In the above methods, the object sentiment analysis model outputs analysis results from multiple sentiment analysis methods, which can make the sentiment analysis results more accurate and reasonable.

[0170] The machine-executable instructions stored in the aforementioned machine-readable storage medium can be used to perform the following operations in the above-mentioned method for analyzing the emotion of an object: the multimedia data is video data containing the target object in a video frame; each video frame in the video data is taken as the current video frame, and the following operations are performed on the current video frame: static facial features are obtained from the current video frame; facial expression change features are obtained from the video frame sequence containing the current video frame; sound features are obtained from the audio data corresponding to the video frame sequence; and language content features are obtained from the audio data and / or subtitle data corresponding to the video frame sequence.

[0171] The machine-executable instructions stored in the aforementioned machine-readable storage medium can be used to perform the following operations in the aforementioned method for analyzing the emotion of an object: obtaining the emotion analysis results corresponding to video frames in video data; determining, based on the emotion analysis results corresponding to the video frames, that there are video frames with sudden emotion changes in the video data; wherein, the emotion analysis results of the video frames with sudden emotion changes are the first analysis results, and the emotion analysis results of the video frames before and after the video frames with sudden emotion changes are the second analysis results; the first analysis results are different from the second analysis results; and based on the second analysis results, smoothing is performed on the emotion analysis results of the video frames with sudden emotion changes to obtain smoothed emotion analysis results.

[0172] This method can avoid obvious errors in sentiment analysis results, thereby improving the accuracy and reasonableness of the results.

[0173] The computer program products of the method, apparatus and electronic device for finding program libraries provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0174] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0175] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0176] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0177] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0178] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An analysis method of an emotion of a subject, characterized by, The method comprises: acquiring multimedia data associated with a target object, and extracting static facial features and dynamic features of the target object from the multimedia data; wherein the dynamic features comprise one or more of expression change features, sound features, and language content features of the target object; inputting the static facial features and the dynamic features into a pre-trained object emotion analysis model, performing feature fusion processing on the static facial features and the dynamic features by the object emotion analysis model to obtain fusion features, and outputting an emotion analysis result of the target object based on the fusion features; The object emotion analysis model comprises at least one dynamic feature processing module, and the dynamic feature processing module is configured to input the dynamic features of a corresponding type; the step of performing feature fusion processing on the static facial features and the dynamic features by the object emotion analysis model to obtain fusion features comprises: for the dynamic feature processing module, inputting the dynamic features into a time sequence model to output context features of the dynamic features; wherein the context features are used to represent the change trend of the features in the dynamic features over time; adding time sequence position information in the context features, and determining the context features added with the time sequence position information as dynamic features to be input into the dynamic feature extraction module; wherein the time sequence position information is used to indicate the position of the features in the context features in time sequence; inputting the dynamic features corresponding to the dynamic feature processing module and the static facial features of the target object into the dynamic feature processing module; performing feature fusion processing on the static facial features and the dynamic features by an attention network in the dynamic feature extraction module to obtain fusion features; wherein the fusion features comprise common key features representing emotions between the static facial features and the dynamic features.

2. The method of claim 1, wherein, The step of extracting the dynamic features of the target object from the multimedia data comprises: if the dynamic features comprise the expression change features, acquiring a specified video frame sequence from video data in the multimedia data; wherein the expression of the target object dynamically changes in the video frame sequence; extracting the expression change features of the target object from the video frame sequence by a pre-trained expression feature extraction model.

3. The method of claim 1, wherein, The step of extracting the dynamic features of the target object from the multimedia data comprises: if the dynamic features comprise the sound features, acquiring a specified audio sequence from audio data in the multimedia data; wherein the audio sequence comprises sound signals emitted by the target object; extracting the sound features of the target object from the audio sequence by a pre-trained sound feature extraction model; wherein the sound features comprise one or more of frequency features, tone features, pitch features, and energy features.

4. The method of claim 1, wherein, The step of extracting the dynamic features of the target object from the multimedia data comprises: if the language content feature is included in the dynamic features, obtaining language content text of the target object from subtitle data and / or audio data in the multimedia data; extracting, by a language content feature extraction model trained in advance, a language content feature of the target object from the language content text; wherein the language content feature is used to represent the language meaning of the language uttered by the target object.

5. The method of claim 1, wherein, The attention network in the dynamic feature processing module includes a self-attention network and a cross-attention network. The step of performing feature fusion processing on the static facial feature and the dynamic feature by the attention network in the dynamic feature extraction module to obtain a fusion feature includes: transforming the dynamic feature to obtain a first input parameter of the self-attention network, inputting the first input parameter into the self-attention network, and outputting an intermediate feature of the dynamic feature; wherein the intermediate feature is used to represent the self-correlation of the dynamic feature; determining a second input parameter of the cross-attention network based on the intermediate feature, determining a third input parameter of the cross-attention network based on the static facial feature, inputting the second input parameter and the third input parameter into the cross-attention network to obtain an output result, and determining the fusion feature based on the output result.

6. The method of claim 5, wherein, The step of determining the second input parameter of the cross-attention network based on the intermediate feature includes: performing first fusion processing on the intermediate feature and the dynamic feature to obtain a first fusion result; and transforming the first fusion result to obtain the second input parameter of the cross-attention network.

7. The method of claim 5, wherein, The step of determining the third input parameter of the cross-attention network based on the static facial feature includes transforming the static facial feature to obtain the third input parameter of the cross-attention network.

8. The method of claim 5, wherein, The step of determining the fusion feature based on the output result includes: performing second fusion processing on the output result and the first fusion result corresponding to the second input parameter to obtain a second fusion result; wherein the second input parameter is obtained by transforming the first fusion result; inputting the second fusion result into a preset first multi-layer perception machine, performing mapping processing on the second fusion result by the first multi-layer perception machine to obtain a mapping result; performing third fusion processing on the mapping result and the second fusion result to obtain the fusion feature.

9. The method of claim 1, wherein, The object emotion analysis model includes a plurality of dynamic feature processing modules, and the dynamic feature processing module is used to input the dynamic feature of a corresponding type and output a fusion feature corresponding to the dynamic feature; The step of outputting the emotion analysis result of the target object based on the fusion feature includes: performing splicing processing on the fusion feature output by the dynamic feature processing module and the static facial feature to obtain a splicing feature; inputting the splicing feature into a preset second multi-layer perception machine to output the emotion analysis result of the target object.

10. The method of claim 9, wherein, The second multi-layer perception machine includes a plurality of branch networks. The step of inputting the spliced features into a preset second multi-layer perception machine to output an emotion analysis result of the target object comprises: The spliced features are input into a plurality of branch networks of the second multi-layer perception machine respectively, wherein the branch networks are preset with feature mapping modes corresponding to the branch networks, and the feature mapping modes comprise one or more of the following: linear combination mapping based on a preset facial action unit, linear combination mapping based on a plurality of preset basic emotion types, linear representation mapping based on positive and negative degrees and intensity degrees of emotions. The branch networks perform mapping processing on the spliced features according to the feature mapping modes corresponding to the branch networks, to obtain emotion analysis results output by the branch networks.

11. The method of claim 1, wherein, The multimedia data is video data containing the target object in a video frame. The step of extracting static facial features and dynamic features of the target object from the multimedia data comprises: Each video frame in the video data is taken as a current video frame one by one, and the following operations are performed on the current video frame: The static facial features are obtained from the current video frame. The expression change features are obtained from a video frame sequence containing the current video frame, the sound features are obtained from audio data corresponding to the video frame sequence, and the language content features are obtained from the audio data and / or subtitle data corresponding to the video frame sequence.

12. The method of claim 11, wherein, After the step of outputting the emotion analysis result of the target object based on the fused features, the method further comprises: Obtaining emotion analysis results of video frames in the video data; Based on the emotion analysis results of the video frames, it is determined that there is an emotion mutation video frame in the video data; wherein the emotion analysis result of the emotion mutation video frame is a first analysis result, and the emotion analysis results of the adjacent video frames before and after the emotion mutation video frame are both second analysis results; the first analysis result is different from the second analysis result; Based on the second analysis result, the emotion analysis result of the emotion mutation video frame is smoothed to obtain a smoothed emotion analysis result.

13. An analysis device of an emotion of a subject, characterized by, The device comprises: A feature extraction module is configured to obtain multimedia data associated with a target object, and extract static facial features and dynamic features of the target object from the multimedia data; wherein the dynamic features comprise one or more of expression change features, sound features, and language content features of the target object. A result output module is configured to input the static facial features and the dynamic features into a pre-trained object emotion analysis model, perform feature fusion processing on the static facial features and the dynamic features through the object emotion analysis model, obtain fused features, and output an emotion analysis result of the target object based on the fused features. The object emotion analysis model comprises at least one dynamic feature processing module, and the dynamic feature processing module is configured to input dynamic features of a corresponding type; and the result output module is further configured to: For the dynamic feature processing module, the dynamic feature corresponding to the dynamic feature processing module and the static facial feature of the target object are input into the dynamic feature processing module; the static facial feature and the dynamic feature are processed by a feature fusion processing of an attention network in the dynamic feature extraction module to obtain a fusion feature; wherein the fusion feature includes common key features representing emotions between the static facial feature and the dynamic feature; The result output module is further configured to: input the dynamic feature into a time sequence model to output a context feature of the dynamic feature; wherein the context feature is used to represent a change trend of a feature in the dynamic feature over time; add time sequence position information to the context feature, and determine the context feature added with the time sequence position information as a dynamic feature to be input into the dynamic feature extraction module; wherein the time sequence position information is used to indicate a position of a feature in the context feature in a time sequence.

14. An electronic device, comprising: A processor and a memory are included, the memory stores machine executable instructions capable of being executed by the processor, and the processor executes the machine executable instructions to implement the object emotion analysis method in any one of claims 1-12.

15. A machine-readable storage medium, characterized in that, The machine readable storage medium stores machine executable instructions, and when the machine executable instructions are called and executed by a processor, the machine executable instructions cause the processor to implement the object emotion analysis method in any one of claims 1-12.

Images