A multimodal co-emotion analysis method, system, storage medium and electronic device

Abstract

The application discloses a multi-modal empathy analysis method and system, a storage medium and an electronic device, relates to the technical field of emotional analysis processing, and extracts various modal features from various sources of to-be-processed multi-modal data, such as language, video, audio, context information, geographical position and the like. Since the information of different modes may be ambiguous, redundant or conflicting, an adaptive mixed modal learning mode is introduced, target language features are generated according to the adaptive mixed modal learning mode and language features of different scales, and target mixed modal features of hyper-modal representation of irrelevant information and conflicting information are suppressed. Cross-modal fusion is performed through the target language features and the target mixed modal features. Since the cross-modal fusion is a complementary relationship between mixed modes, which is implicitly inferred fusion operation, the fusion operation realizes robust and accurate emotional prediction, thereby effectively performing multi-modal emotional analysis and improving the effect and accuracy of multi-modal empathy emotional analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sentiment analysis processing, more particularly, to a multi-modal empathy analysis method and system, a storage medium and an electronic device. BACKGROUND

[0002] Multi-modal empathy sentiment analysis is an important branch of artificial intelligence, and has a very wide range of applications. In the process of multi-modal empathy sentiment analysis, by enabling the model to understand emotions, the humanization level of dialogue systems and intelligent agents can be improved, and user experience and interaction effects can be improved.

[0003] Existing multi-modal empathy sentiment analysis mainly relies on manually designed features, such as text features based on words, syntax, or dictionary-based sentiment features, and visual features such as expression categories and body postures. These features from different modalities are simply combined together, and traditional machine learning models are used to classify these combined features to achieve multi-modal empathy sentiment analysis. Such a way is limited by the quality and effectiveness of feature design, and it is difficult to capture complex inter-modal interactions and deep semantics, resulting in poor multi-modal empathy sentiment analysis results and low accuracy.

[0004] Therefore, how to improve the effect and accuracy of multi-modal empathy sentiment analysis is a problem that needs to be solved by the present application. SUMMARY

[0005] Therefore, the present application discloses a multi-modal empathy analysis method, system, storage medium and electronic device, which aims to improve the effect and accuracy of multi-modal empathy sentiment analysis.

[0006] In order to achieve the above-mentioned purpose, the disclosed technical solution is as follows:

[0007] The first aspect of the present application discloses a multi-modal empathy analysis method, which comprises:

[0008] Obtaining multi-modal data to be processed, and extracting each modal feature from the multi-modal data to be processed;

[0009] Uniformizing each modal feature to obtain uniform features of each modal;

[0010] Defining the uniform features of each modal as first-scale language features; wherein the first-scale language features are the base layer of a multi-scale feature pyramid;

[0011] Generating a preset scale semantic feature through the first-scale language feature;

[0012] generate a target language feature and a target mixed modal feature according to an adaptive mixed modal learning manner, the first scale language feature, and the preset scale semantic feature; wherein the target mixed modal feature is a hyper-modal representation learned from multi-dimensional features to suppress irrelevant information and suppress conflicting information;

[0013] perform cross-modal fusion through the target language feature and the target mixed modal feature to obtain a joint multi-modal representation; the cross-modal fusion is a fusion operation in which a complementary relationship between mixed modalities is implicitly inferred;

[0014] perform empathetic sentiment analysis through the joint multi-modal representation to obtain an empathetic sentiment analysis output result.

[0015] Preferably, the feature uniformization of each modal feature to obtain uniform features of each modality includes:

[0016] randomly initializing a token vector for each modal feature;

[0017] embedding each modal feature into the initialized token vector through a feature extractor composed of a parallel full connection layer and a transformer layer to obtain a spliced uniform dimension;

[0018] inputting the spliced uniform dimension into the transformer layer through the full connection layer to obtain the uniform features of each modality.

[0019] Preferably, the preset scale semantic feature includes a second scale language feature and a third scale language feature, and the generation of the preset scale semantic feature through the first scale language feature includes:

[0020] taking the first scale language feature as a basic layer of a multi-scale feature pyramid, and generating the second scale language feature and the third scale language feature according to the basic layer and a plurality of transformer layers;

[0021] wherein the second scale language feature is used to capture sentence-level semantics, and the third scale language feature is used to capture chapter-level semantics.

[0022] Preferably, the generation of the target language feature and the target mixed modal feature according to the adaptive mixed modal learning manner, the first scale language feature, and the preset scale semantic feature includes:

[0023] calculating a similarity matrix between the first scale language feature and the preset scale semantic feature and a remaining modal feature through multi-head attention; wherein the remaining modal feature is a modal feature other than text; performing product calculation on the remaining modal feature with the similarity matrix as a weight to obtain a weighted remaining modal feature;

[0024] adding the weighted remaining modal feature and the mixed modal feature of the previous layer in the adaptive mixed modal learning layer group and iteratively updating to obtain the mixed modal feature of the current layer in the adaptive mixed modal learning layer group;

[0025] In the process of multi-modal fusion and prediction, the initialized token vector is spliced with the first scale language feature and the preset scale semantic feature to obtain a target language feature, and the initialized token vector is spliced with the mixed modal feature of the current layer to obtain a target mixed modal feature.

[0026] Preferably, the cross-modal fusion based on the target language feature and the target mixed modal feature to obtain a joint multi-modal representation comprises:

[0027] The target language feature is taken as a query, and the target mixed modal feature is taken as a key and a value.

[0028] The query, the key and the value are fused to obtain a joint multi-modal representation.

[0029] Preferably, the joint multi-modal representation is used for empathy sentiment analysis to obtain an empathy sentiment analysis output result, which comprises:

[0030] The cross-entropy loss is taken as a loss function.

[0031] The output of the cross-modal fusion converter is applied to a classifier, the loss function and the joint multi-modal representation to obtain an empathy sentiment analysis output result.

[0032] The second aspect of the present application discloses a multi-modal empathy analysis system, which comprises:

[0033] An acquisition unit is configured to acquire multi-modal data to be processed and extract modal features from the multi-modal data to be processed.

[0034] A feature uniformization unit is configured to perform feature uniformization on the modal features to obtain uniform features of the modal features.

[0035] A definition unit is configured to define the uniform features of the modal features as first scale language features, wherein the first scale language features are base layers of a multi-scale feature pyramid.

[0036] A first generation unit is configured to generate preset scale semantic features based on the first scale language features.

[0037] The second generation unit is configured to generate target language features and target mixed-mode features according to the adaptive mixed-mode learning manner, the first scale language features, and the preset scale semantic features; the target mixed-mode features are hyper-modal representations learned from multi-dimensional features to suppress irrelevant information and suppress conflict information.

[0038] The cross-modal fusion unit is configured to perform cross-modal fusion on the target language features and the target mixed-mode features to obtain joint multi-modal representations; the cross-modal fusion is a fusion operation in which complementary relationships between mixed modes are implicitly inferred.

[0039] The empathetic emotion analysis unit is configured to perform empathetic emotion analysis on the joint multi-modal representations to obtain an empathetic emotion analysis output result.

[0040] Preferably, the feature unification unit comprises:

[0041] The random initialization module is configured to randomly initialize token vectors for the modal features.

[0042] The embedding module is configured to embed the modal features into the initialized token vectors by a feature extractor composed of parallel full connection layers and transformer layers to obtain a unified dimension after splicing.

[0043] The input acquisition module is configured to input the unified dimension after splicing into the transformer layer by a full connection layer to obtain unified features of the modal.

[0044] The third aspect of the present application discloses a storage medium, the storage medium comprising stored instructions, wherein the instructions, when executed, control a device in which the storage medium is located to perform the multi-modal empathetic analysis method according to any one of the first aspect.

[0045] The fourth aspect of the present application discloses an electronic device comprising a memory and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by one or more processors to perform the multi-modal empathetic analysis method according to any one of the first aspect.

[0046] The technical scheme can know that the application discloses a multi-modal empathy analysis method, system, storage medium and electronic device, obtains multi-modal data to be processed, extracts modal features from the multi-modal data to be processed, unifies the modal features, obtains unified features of each modal, defines the unified features of each modal as first scale language features, wherein the first scale language features are a basic layer of a multi-scale feature pyramid, generates preset scale semantic features through the first scale language features, generates target language features and target mixed modal features according to an adaptive mixed modal learning mode, the first scale language features and the preset scale semantic features; wherein the target mixed modal features are a hyper-modal representation for learning to suppress irrelevant information and suppress conflict information from multi-dimensional features, and joint multi-modal representation is obtained through cross-modal fusion of the target language features and the target mixed modal features; the cross-modal fusion is a fusion operation in which the complementary relationship between the mixed modalities is implicitly inferred, and empathy sentiment analysis output results are obtained through empathy sentiment analysis based on the joint multi-modal representation.

[0047] The application has the following beneficial effects: multi-modal data to be processed is extracted from various sources, such as language, video, audio, context information, user basic information, geographic location and the like, modal features are extracted from the multi-modal data to be processed, the adaptive mixed modal learning mode is introduced, target language features are generated according to the adaptive mixed modal learning mode and different scale language features, and target mixed modal features of a hyper-modal representation for learning to suppress irrelevant information and suppress conflict information are generated, cross-modal fusion is performed through the target language features and the target mixed modal features, the cross-modal fusion is a fusion operation in which the complementary relationship between the mixed modalities is implicitly inferred, the fusion operation realizes robust and accurate sentiment prediction, and therefore multi-modal sentiment analysis is effectively performed, and the effect and accuracy of multi-modal empathy sentiment analysis are improved. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute the embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0049] Figure 1 A flowchart of a multi-modal empathy analysis method disclosed in an embodiment of the application;

[0050] Figure 2 A flowchart of obtaining unified features of each modal disclosed in an embodiment of the application;

[0051] Figure 3A flowchart of a process for generating target language features and target mixed modality features disclosed in an embodiment of the present application is shown in FIG. 1.

[0052] Figure 4 A flowchart of a process for obtaining joint multi-modal representation disclosed in an embodiment of the present application is shown in FIG. 2.

[0053] Figure 5 A flowchart of a process for obtaining empathetic sentiment analysis output result disclosed in an embodiment of the present application is shown in FIG. 3.

[0054] Figure 6 A structural diagram of a multi-modal empathetic analysis system disclosed in an embodiment of the present application is shown in FIG. 4.

[0055] Figure 7 A structural diagram of an electronic device disclosed in an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0057] In the present application, the term “comprising” or “including” or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement “comprising a” does not exclude the presence of another identical element in the process, method, article or device including the element.

[0058] As known from the background, the existing multi-modal empathetic sentiment analysis is limited by the quality and effectiveness of feature design, and it is difficult to capture complex inter-modal interaction and deep semantics, resulting in poor effect and low accuracy of multi-modal empathetic sentiment analysis.

[0059] To solve the above problems, the application discloses a multi-modal empathy analysis method, system, storage medium and electronic device, which extracts various modal features from various sources of to-be-processed multi-modal data, such as language, video, audio, context information, user basic information, geographic location and the like. Since the information of different modalities may be ambiguous, redundant or conflicting, an adaptive hybrid multi-modal learning method is introduced, target language features are generated according to the adaptive hybrid multi-modal learning method and language features of different scales, and target hybrid multi-modal features of hyper-modal representation that suppresses irrelevant information and conflicting information are generated. Cross-modal fusion is performed through the target language features and the target hybrid multi-modal features. Since the cross-modal fusion is a complementary relationship between hybrid modalities that is implicitly inferred fusion operation, the fusion operation realizes robust and accurate emotion prediction, thereby effectively performing multi-modal emotion analysis and improving the effect and accuracy of multi-modal empathy emotion analysis. The specific implementation manner is specifically described in the following embodiments.

[0060] It should be noted that the multi-modal empathy analysis method, system, storage medium and electronic device provided by the application can be used in the fields of emotion analysis, deep learning, computer vision, multi-modal information processing and the like. The above is only an example and does not limit the application of the multi-modal empathy analysis method, system, storage medium and electronic device provided by the application.

[0061] Reference Figure 1 As shown in the figure, it is a flowchart of a multi-modal empathy analysis method disclosed by the embodiments of the application. The multi-modal empathy analysis method mainly includes the following steps:

[0062] S101: acquiring to-be-processed multi-modal data and extracting various modal features from the to-be-processed multi-modal data.

[0063] The to-be-processed multi-modal data is a multi-modal input, and the to-be-processed multi-modal data is composed of language, audio, visual information, geographic location, user basic information, context information and the like.

[0064] The language can be text information input through an external device such as a keyboard.

[0065] The audio is the sound input by the user through a device such as a microphone after the system obtains the microphone permission; the visual information can capture the video of the user through a visual acquisition device such as a camera; the geographic location information is the GPS longitude and latitude coordinates obtained after the location permission is turned on; the user basic information is the personal identity portrait information filled in by the user when logging in, including but not limited to gender, age, hobbies and the like; the context information includes but is not limited to the identification of other users around, system time, sensor temperature and the like surrounding scene and background.

[0066] For audio information, audio signals can be processed using a PyTorch-based Torchaudio audio processing library or the like to obtain audio representation vectors. The specific process of processing audio signals based on the PyTorch-based Torchaudio audio processing library or the like to obtain audio representation vectors is as follows:

[0067] First, the original audio signal is loaded for resampling to a uniform sampling rate, and then a time-frequency conversion technique (such as short-time Fourier transform) is used to convert the time-domain signal to a spectral representation;

[0068] Then, a Mel filter bank is used to generate a Mel spectrum and perform logarithmic compression and standardization processing, and high-level acoustic features are extracted through multi-layer convolution and pooling operations, outputting a 768-dimensional audio representation vector. The entire process converts the original audio into a compact vector representation that preserves speech content, speaker characteristics, and emotional information.

[0069] For text information and visual information, image encoders based on the ViT-B / 32 architecture and text encoders based on the masked self-attention transformer converter architecture can be used to unify the representation and align the image and text data.

[0070] The specific process of using the ViT-B / 32 architecture image encoder and the masked self-attention transformer converter architecture text encoder to unify the representation and align the image and text data is as follows:

[0071] (1) The input image is divided into 32x32 pixel blocks, linearly projected into embedding vectors and added with position encoding, processed through 12 layers of Transformer blocks (each layer contains multi-head self-attention mechanism and MLP), and finally the 768-dimensional vector corresponding to the [CLS] token is extracted as the image representation;

[0072] (2) The text data input is tokenized by the masked self-attention transformer converter architecture text encoder and added with position encoding, and processed through 12 layers of masked self-attention Transformer blocks (each layer calculates context-aware representation, and the masking mechanism ensures that the current position only depends on the left context), and finally the 768-dimensional vector of the [CLS] token is extracted as the text representation;

[0073] (3) Through a contrastive learning objective function (InfoNCE loss), optimization is performed in a shared embedding space to maximize the cosine similarity of matching image-text pair features and minimize non-matching pairs, achieving cross-modal alignment.

[0074] For heterogeneous data such as geographic location, user basic information, and context information, the digital data is first decoded, for example, the accurate position coordinates (31.8206417, 117.1262532) obtained by using GPS are translated into Hefei, Anhui Province, China by reverse geocoding, and these different information is integrated as a piece of text by using heuristic prompt learning. Specifically, a heuristic prompt P = "the current location is <geographic location>, the current time is <time information>, and the following information is included: <context information>, and the user's personal information is as follows: <basic information>".

[0075] S102: Uniformize the features of each modality to obtain uniform features of each modality.

[0076] The process of uniformizing the features of each modality to obtain uniform features of each modality is specifically as shown in Figure 2 , and mainly includes the following steps:

[0077] S201: Randomly initialize a token vector for each modality feature.

[0078] In S201, a low-dimensional token vector is randomly initialized for each modality, and the low-dimensional token vector is represented by , the low-dimensional token vector is a 128-dimensional learnable token, m e {l, v, a}, where l represents language, v represents video, and a represents audio; each modality feature in the above is represented by .

[0079] S202: Embed each modality feature into the initialized token vector through the feature extractor composed of parallel fully connected layers and transformer layers to obtain the uniform dimension after splicing.

[0080] For the above multi-modal input, the features of each modality are unified by introducing parallel fully connected layers and transformer layers, specifically, a low-dimensional token (i.e. low-dimensional token vector) is randomly initialized for each modality, the basic modality information is embedded into the initialized low-dimensional token vector using the transformer layer, and the uniform features of each modality are obtained through the modality feature extractor and the corresponding parameters and splicing operation.

[0081] The corresponding parameters of the modality feature extractor are represented by .

[0082] S203: Input the uniform dimension after splicing into the transformer layer through the fully connected layer to obtain the uniform features of each modality.

[0083] In S203, the modal feature extractor composed of a parallel full connection layer and a transformer layer is used to splice the initialized token vector and each modal feature (i.e., the projected feature), and then project to a unified dimension through a full connection layer and input to a Transformer layer, and finally obtain the unified feature of each modal, i.e., a 128-dimensional vector. . The expression is shown in formula (1).

[0084] (1)

[0085] wherein, is the modal feature extractor; is the low-dimensional token vector; is each modal feature in the above; is the corresponding parameter of the modal feature extractor.

[0086] The embodiment embeds necessary modal information into the initialized low-dimensional token vector, which is beneficial to reduce redundant information irrelevant to human emotion, so as to realize higher efficiency with fewer parameters.

[0087] S103: define the unified feature of each modal as the first scale language feature; wherein the first scale language feature is the basic layer (128-dimensional vector) of the multi-scale feature pyramid.

[0088] It should be noted that the first scale language feature is a low scale language feature. The low scale language feature refers to the text modal unified feature (128-dimensional vector) directly output from the modal embedding stage, which has not yet undergone high-order feature abstraction of the subsequent Transform layer. It retains complete details and contains full-granularity information of basic language elements such as words and grammar.

[0089] The low scale language feature is the original language feature, which serves as the "origin" and "detail preserver". It is both the starting point of multi-scale expansion and the semantic detail guarantee of the fusion process.

[0090] S104: generate a preset scale semantic feature through the first scale language feature.

[0091] In S104, the first scale language feature is taken as the basic layer of the multi-scale feature pyramid, and according to the basic layer and two transformer layers, the second scale language feature (medium scale language feature) and the third scale language feature (high scale language feature) are generated respectively.

[0092] The preset scale semantic features include second scale language features and third scale language features. The second scale language features are used to capture sentence-level semantics (phrase-level semantics); and the third scale language features are used to capture chapter-level semantics.

[0093] The second scale language features and the third scale language features retain modality specificity, as pure text features that are not contaminated by other modalities, forming a gradient path of "low scale → middle scale → high scale". Positioning the first scale language features as low scale language features essentially establishes a feature evolution path from the concrete to the abstract. This not only retains the original semantic integrity (avoids information contamination caused by premature fusion), but also realizes precise screening of emotion-related features through multi-scale abstraction, ultimately improving the sensitivity of the model to multi-modal fusion guided by language.

[0094] The core role of the first scale language features is to build a multi-scale feature pyramid as the basic layer (scale 1) of the language feature pyramid, and the subsequent transform layers generate the preset scale semantic features based on this. The first scale language features provide basic information for the construction of the second scale language features and the third scale language features, and subsequent transformer layers further model in the time and semantic dimensions to capture longer-range and more abstract dependencies. The fine-grained semantics and emotional cues in the first scale language features can be involved in the adaptive multi-modal learning layer to help the model retain key emotional details when fusing visual and audio information, reducing the risk of losing details in multi-modal interaction.

[0095] It should be noted that for the construction of the preset scale semantic features, the features of the language modality obtained in the previous step are defined as low scale language features, and two transform layers are introduced to learn middle scale language features and high scale language features. Each transform layer can use 8 attention heads to model the information of each modality. Unlike the transformer layer that transmits basic information to the initialized tokens in the above-mentioned modality embedding stage, in the modeling stage, the first scale language features, the second scale language features, and the third scale language features are modeled through the transformer layer.

[0096] S105: generating target language features and target mixed modal features according to the adaptive mixed modal learning manner, the first scale language features, and the preset scale semantic features; wherein the target mixed modal features are super-modal representations learned from multi-dimensional features to suppress irrelevant information and suppress conflicting information.

[0097] In S105, the adaptive mixed modal learning module is used to learn refined target mixed modal representations that contain irrelevant / conflicting information and highly complementary language features.

[0098] The target mixed-modal representation is a vector. The process of obtaining the vector is as follows:

[0099] 1) the guiding role of language features as queries;

[0100] 2) the filtering of conflicting information by similarity weights;

[0101] 3) the iterative updating mechanism of super-modal features Irrelevant / conflicting information is irrelevant or even contradictory, and will be suppressed in the adaptive mixed-modal learning process. Highly complementary language features are key information that can complement and strengthen each other and other modalities, and these features will be highlighted and enhanced in the fusion process. The calculation formula of is shown in formula (2).

[0102] (2)

[0103] wherein j∈{1, 2, 3}; is the output of the j-1th adaptive mixed-modal learning layer; and are learnable parameters, and the dimension is 128x16; is the similarity matrix between the speech features and the audio features; is the similarity matrix between the language modalities and the visual modalities; is the weighted audio features; is the weighted visual features; and are the unified features of each modality obtained by formula (1), a is audio; v is video.

[0104] The expression of is shown in formula (3).

[0105] (3)

[0106] wherein, is the query query of the multi-head attention mechanism; is the key key of the multi-head attention mechanism; as the query; is a learnable parameter matrix, and the dimension is 128x16; is the transpose of the learnable parameter matrix, and the dimension is 16x128; is the transpose of the representation of the visual modality; is the dimension of each attention head, and in practice, 8 attention heads are used, and is set to 16.

[0107] The expression is shown as formula (4).

[0108] (4)

[0109] wherein, is the transpose of the learnable parameter, and the dimension is 16x128; is the transpose of the representation of the previous visual modality.

[0110] The adaptive multi-modal learning module is composed of two transform layers and an adaptive multi-modal learning layer group, and the purpose is to learn language features of different scales. Under the guidance of language features, the target multi-modal features are adaptively learned from multi-dimensional features, i.e., visual modalities and audio modalities. The adaptive multi-modal learning layer group includes at least three adaptive multi-modal learning layers.

[0111] The specific process of generating target language features and target multi-modal features is shown in FIG. 4, and mainly includes the following steps: Figure 3

[0112] S301: Calculate the similarity matrix between different scale language features (i.e., first scale language features and preset scale semantic features) and remaining modal features through multi-head attention; wherein, the remaining modal features are other modal features (such as visual modalities, audio modalities) except text.

[0113] In S301, in the three adaptive multi-modal learning layers in the adaptive multi-modal learning layer group, for language features of different scales, first, an hyper-modal feature is initialized; wherein, the hyper-modal feature is a self-defined 128-dimensional learnable vector used to represent a feature vector fused with each modality, i.e., a multi-modal feature.

[0114] Secondly, the relationship between the obtained language features (i.e., first scale language features, second scale language features and third scale language features) and the remaining modalities is calculated through multi-head attention, and the initialized hyper-modal feature is updated through the relationship; wherein, the process of calculating the language features through multi-head attention is consistent with the process of learning the language features of the second scale language features and the third scale language features by the two transform layers described above. Each transform layer uses multi-head attention (such as 8-head attention) to model the information of each modality.

[0115] ​Then, the extracted language features of different scales (i.e., the first-scale language feature, the second-scale language feature, and the third-scale language feature) are used as queries, and the remaining modal features are used as keywords to obtain a similarity matrix between the language features and the remaining modal features.

[0116] S302: The similarity matrix is multiplied with the remaining modal features as weights to obtain weighted remaining modal features.

[0117] S303: The weighted remaining modal features are added to the mixed modal features of the previous layer in the adaptive mixed modal learning layer group and iteratively updated to obtain the mixed modal features of the current layer in the adaptive mixed modal learning layer group.

[0118] In S303, the weighted remaining modal features are added to the updated hyper-modal features of the previous layer and iteratively updated to obtain the hyper-modal features of the current layer in the adaptive mixed modal learning layer group.

[0119] S304: In the process of multi-modal fusion and prediction, the initialized token vector is concatenated with the first-scale language feature and the preset-scale semantic feature to obtain target language features, and the initialized token vector is concatenated with the mixed modal features of the current layer to obtain target mixed modal features.

[0120] In the process of multi-modal fusion and prediction, a token vector (i.e., token token) is first initialized, and the initialized token vector is connected with language features of different scales (such as hyper-modal features of each layer of the adaptive mixed modal learning layer group, first-scale language features, second-scale language features, and third-scale language features) to obtain a new language feature (target language feature) and a new hyper-modal feature (target mixed modal feature).

[0121] The token vector is used to capture global information at the head of the sequence.

[0122] S106: Cross-modal fusion is performed on the target language features and the target mixed modal features to obtain joint multi-modal representations; cross-modal fusion is a fusion operation in which the complementary relationship between mixed modalities is implicitly inferred.

[0123] In S106, the joint representation and the complementary information are transmitted to the initialized token vector by applying a cross-modal fusion transformer.

[0124] The cross-modal fusion transformer is a deep stack of multiple cross-modal attention blocks, with a depth of M.

[0125] The complementary information refers to the first-scale language feature , the second-scale language feature and the third scale language feature . The different scale language features are of size 128 dimensions, and the different scale language features include the first scale language feature , the second scale language feature and the third scale language feature . The expression of the different scale language features is shown in formula (5).

[0126] (5)

[0127] wherein, is the i-th layer of the language feature learning Transformer; is the parameter of the corresponding Transformer layer of the i-th layer of the language feature learning Transformer.

[0128] The joint representation is represented by . The complementary information is represented by .

[0129] The joint multi-modal representation is represented by formula (6).

[0130] (6)

[0131] wherein, is the connection of the initialized token vector and the third scale language feature; is the initialized token vector; is the third scale language feature; is the connection of the initialized token vector and the joint representation; is the joint representation; is the joint multi-modal representation.

[0132] The joint multi-modal representation is specifically obtained, as shown in Figure 4 , mainly including the following steps:

[0133] S401: Taking the target language feature as a query, and taking the target mixed modal feature as a key and a value.

[0134] S402: Fusing the query and the key and the value to obtain the joint multi-modal representation.

[0135] Specifically, the cross-modal fusion transformer takes the target language feature as a query, and fuses the target mixed modal feature as a key and a value to obtain the joint multi-modal representation.

[0136] S107: Performing the empathetic sentiment analysis through the joint multi-modal representation to obtain an empathetic sentiment analysis output result.

[0137] Specifically, the co-empathy sentiment analysis output result is obtained, as shown in Figure 5 includes the following steps:

[0138] S501: Cross-entropy loss is taken as a loss function.

[0139] The loss function of the present application includes but is not limited to cross-entropy loss.

[0140] S502: The output of the cross-modal fusion converter is applied to the classification, the loss function and the joint multi-modal representation to obtain the co-empathy sentiment analysis output result.

[0141] The output application classifier is composed of a linear layer and a softmax layer.

[0142] The present application improves the performance of sentiment analysis by solving the adverse effects of destructive information and irrelevant information in mixed modalities such as vision and audio, and can comprehensively identify human emotional attitudes from various types of data, which has a significant advantage in improving the understanding of human emotions. Specifically, in the adaptive mixed modality learning module designed in the present application, the language features of different scales are taken as the query of multi-head attention, the similarity matrix between language features and other modal features is calculated, the information with high similarity is retained, and the destructive information and irrelevant information with low similarity to emotional text are removed, solving the adverse effects of such interference information in visual and audio modalities.

[0143] Since the information from different modalities may be ambiguous, redundant or conflicting, such as strong light in video, interference information in picture background and background noise in speaker audio, etc., which has an adverse effect on achieving better sentiment understanding performance. Therefore, each modality of the present application is first converted into a unified form by using a transformer converter with an initialization mark. This operation not only suppresses the redundant information across modalities, but also compresses the length of the long sequence to facilitate efficient model calculation. Then, an adaptive mixed modality learning module is introduced, which uses language features of different scales to guide visual and audio modalities, producing intermediate mixed modality representations containing less sentiment-irrelevant information. Finally, a cross-modal fusion module is designed, which takes language features, mixed modality features as query, key and value respectively through the cross-modal fusion module, and the complementary relationship between language, vision, audio, and other mixed modalities is implicitly inferred, realizing robust and accurate sentiment prediction.

[0144] The present application obtains rich information from various sources such as language, video, audio, context information, user basic information, geographical location, etc. Since the information from different modalities may be ambiguous, redundant or conflicting, the present application introduces an adaptive mixed modality learning module to learn a hyper-modal representation that can effectively suppress irrelevant / conflicting information from visual, auditory and other features under the guidance of language features of different scales. Using the obtained hyper-modal representation, the model can obtain a complementary joint representation through multi-modal fusion, thereby realizing effective multi-modal sentiment analysis and significantly improving the effect and accuracy in multi-modal sentiment understanding.

[0145] The beneficial effects of the embodiments of the present application are as follows: various modal features are extracted from various sources of to-be-processed multi-modal data such as language, video, audio, context information, user basic information, geographical location and the like. Since the information from different modalities may be ambiguous, redundant or conflicting, an adaptive mixed modality learning method is introduced to generate target language features and target mixed modality features of hyper-modal representation that suppresses irrelevant information and conflicting information according to the adaptive mixed modality learning method and language features of different scales. Cross-modal fusion is performed through the target language features and the target mixed modality features. Since the cross-modal fusion is a fusion operation in which the complementary relationship between mixed modalities is implicitly inferred, the fusion operation realizes robust and accurate sentiment prediction, thereby effectively performing multi-modal sentiment analysis and improving the effect and accuracy of multi-modal empathetic sentiment analysis. Based on the multi-modal empathetic analysis method, the present application can construct a more intelligent and humanized interactive system that can perceive and understand the emotional needs of users and provide more empathetic feedback.

[0146] Based on the above embodiments Figure 1 A multi-modal empathetic analysis method is disclosed. The embodiments of the present application also disclose a multi-modal empathetic analysis system, as shown in Figure 6 The multi-modal empathetic analysis system comprises:

[0147] An acquisition unit 601 is configured to acquire to-be-processed multi-modal data and extract various modal features from the to-be-processed multi-modal data.

[0148] A feature unification unit 602 is configured to perform feature unification on the various modal features to obtain unified features of the various modalities.

[0149] A definition unit 603 is configured to define the unified features of the various modalities as first-scale language features. The first-scale language features are the base layer of a multi-scale feature pyramid.

[0150] A first generation unit 604 is configured to generate preset-scale semantic features through the first-scale language features.

[0151] The second generation unit 605 is configured to generate target language features and target mixed-mode features according to the adaptive mixed-mode learning manner, the first scale language features, and preset scale semantic features; the target mixed-mode features are super-modal representations learned from multi-dimensional features to suppress irrelevant information and suppress conflict information.

[0152] The cross-modal fusion unit 606 is configured to perform cross-modal fusion on the target language features and the target mixed-mode features to obtain joint multi-modal representations; the cross-modal fusion is a fusion operation in which complementary relationships between mixed modes are implicitly inferred.

[0153] The empathetic emotion analysis unit 607 is configured to perform empathetic emotion analysis on the joint multi-modal representations to obtain an empathetic emotion analysis output result.

[0154] Further, the feature unification unit 602 includes:

[0155] The random initialization module is configured to randomly initialize token vectors for each modality feature.

[0156] The embedding module is configured to embed each modality feature into the initialized token vectors by a feature extractor composed of parallel full connection layers and transformer layers, to obtain a unified dimension after splicing.

[0157] The input acquisition module is configured to input the unified dimension after splicing into the transformer layer through a full connection layer, to obtain unified features of each modality.

[0158] Further, the preset scale semantic features include second scale language features and third scale language features, and the first generation unit 604 is specifically configured to take the first scale language features as a basic layer of a multi-scale feature pyramid, and generate the second scale language features and the third scale language features according to the basic layer and a plurality of transformer layers; the second scale language features are used to capture sentence-level semantics, and the third scale language features are used to capture chapter-level semantics.

[0159] Further, the second generation unit 605 includes:

[0160] The first calculation module is configured to calculate a similarity matrix between the first scale language features and the preset scale semantic features and remaining modality features by multi-head attention; the remaining modality features are modality features other than text.

[0161] The second calculation module is configured to perform product calculation on the similarity matrix as a weight and the remaining modality features, to obtain weighted remaining modality features.

[0162] The iterative updating module is configured to add the weighted residual modal feature to the mixed modal feature of the previous layer in the adaptive mixed modal learning layer group and perform iterative updating to obtain the mixed modal feature of the current layer in the adaptive mixed modal learning layer group.

[0163] The splicing module is configured to splice the initialized token vector with the first-scale language feature and the preset-scale semantic feature to obtain a target language feature, and splice the initialized token vector with the mixed modal feature of the current layer to obtain a target mixed modal feature in the process of multi-modal fusion and prediction.

[0164] Further, the cross-modal fusion unit 606 comprises:

[0165] The first determination module is configured to take the target language feature as a query and take the target mixed modal feature as a key and a value.

[0166] The fusion module is configured to fuse the query, the key and the value to obtain a joint multi-modal representation.

[0167] Further, the empathic sentiment analysis unit 607 comprises:

[0168] The second determination module is configured to take the cross-entropy loss as a loss function.

[0169] The acquisition module is configured to apply a classifier, a loss function and the joint multi-modal representation to an output of the cross-modal fusion converter to obtain an empathic sentiment analysis output result.

[0170] The embodiments of the present application have the following beneficial effects: various modal features are extracted from various sources of to-be-processed multi-modal data such as language, video, audio, context information, user basic information and geographic location information. Since the information of different modalities may be ambiguous, redundant or conflicting, an adaptive mixed modal learning manner is introduced to generate a target language feature according to the adaptive mixed modal learning manner and different-scale language features, and a target mixed modal feature of a hyper-modal representation that suppresses irrelevant information and conflicting information. Cross-modal fusion is performed through the target language feature and the target mixed modal feature. Since the cross-modal fusion is a complementary relationship between mixed modalities that is implicitly inferred fusion operation, the fusion operation realizes robust and accurate sentiment prediction, thereby effectively performing multi-modal sentiment analysis and improving the effect and accuracy of multi-modal empathic sentiment analysis.

[0171] The embodiments of the present application also provide a storage medium comprising stored instructions, wherein the instructions control a device where the storage medium is located to perform the multi-modal empathic analysis method as described above when the instructions are executed.

[0172] The embodiments of the present application also provide an electronic device, a structural schematic diagram of which is as shown inFigure 7 As shown, the computer system 700 includes a memory 701, and one or more processors 703, where the one or more instructions 702 are stored in the memory 701 and configured to be executed by the one or more processors 703 to perform the above-mentioned multi-modal co-empathic analysis method.

[0173] For each method embodiments described above, for simplicity of description, the method is described as a series of acts but those of skill in the art will appreciate that the acts described can be performed in other sequences or even concurrently, and that described embodiments can be combined with other embodiments.

[0174] It should be noted that each of the above-described examples of the present disclosure is described as a progressive manner, and each example focuses on the differences from other examples. The same or similar parts between examples can be mutually referred to. For the device examples, since they are basically similar to the method examples, the description is relatively simple, and the relevant parts can be referred to the description of the method examples.

[0175] The steps in the method of each embodiment of the present disclosure can be adjusted in sequence, combined, and reduced according to actual needs.

[0176] Finally, it should be noted that in this document, relationship terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0177] The above description of disclosed embodiments enables a person skilled in the art to implement or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0178] The above is only the preferred embodiment of the present disclosure, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present disclosure, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present disclosure.

Claims

1. A multimodal empathy analysis method, characterized in that, The method includes: Acquire the multimodal data to be processed, and extract each modal feature from the multimodal data to be processed; Feature unification is performed on the features of each modality to obtain the unified features of each modality; The unified features of each modality are defined as first-scale language features; wherein, the first-scale language features are the base layer of the multi-scale feature pyramid; Based on the first-scale language features, a preset-scale semantic feature is generated, wherein the preset-scale semantic feature includes a second-scale language feature and a third-scale language feature, the second-scale language feature is used to capture sentence-level semantics, and the third-scale language feature is used to capture discourse-level semantics; Based on the adaptive mixed modality learning method, the first-scale language features, and the preset-scale semantic features, target language features and target mixed modality features are generated; wherein, the target language features are composed of an initialized token vector, the first-scale language features, and the preset-scale semantic features, and the target mixed modality features are composed of the initialized token vector and the mixed modality features of the current layer, and the target mixed modality features are updated by using the relationship between the language features obtained by multi-head attention computation and the remaining modality features; Cross-modal fusion is performed using the target language features and target mixed modal features to obtain a joint multimodal representation; the cross-modal fusion is a fusion operation in which the complementary relationships between mixed modalities are implicitly inferred. Empathic sentiment analysis is performed using the joint multimodal representation to obtain the empathic sentiment analysis output results.

2. The method according to claim 1, characterized in that, The process of unifying the features of each modality to obtain unified features for each modality includes: Randomly initialize the token vector for each modality feature; A feature extractor consisting of parallel fully connected layers and transformer layers embeds the features of each modality into the initial token vector to obtain a unified dimension after concatenation. By using a fully connected layer, the concatenated unified dimension is input into the transformer layer to obtain the unified features of each modality.

3. The method according to claim 1, characterized in that, The step of generating semantic features of a preset scale using the language features of the first scale includes: The first-scale language features are used as the base layer of the multi-scale feature pyramid, and second-scale and third-scale language features are generated based on the base layer and multiple transformer layers.

4. The method according to claim 1, characterized in that, The step of generating target language features and target mixed-modality features based on the adaptive mixed-modality learning method, the first-scale language features, and the preset-scale semantic features includes: The similarity matrix between the first-scale language features and the preset-scale semantic features and the remaining modal features is calculated using multi-head attention; wherein, the remaining modal features are modal features other than text. The similarity matrix is ​​used as a weight to multiply with the remaining modal features to obtain weighted remaining modal features; The weighted residual modal features are added to the hybrid modal features of the previous layer in the adaptive hybrid modal learning layer group and iteratively updated to obtain the hybrid modal features of the current layer in the adaptive hybrid modal learning layer group; In the process of multimodal fusion and prediction, the initialized token vector is concatenated with the first-scale language features and the preset-scale semantic features to obtain the target language features, and the initialized token vector is concatenated with the mixed modal features of the current layer to obtain the target mixed modal features.

5. The method according to claim 1, characterized in that, The process of cross-modal fusion of the target language features and target mixed modal features to obtain a joint multimodal representation includes: The target language features are used as the query, and the target mixed modality features are used as the key and value; The query, along with the key and value, are fused to obtain a joint multimodal representation.

6. The method according to claim 1, characterized in that, The empathy sentiment analysis performed through the joint multimodal representation to obtain the empathy sentiment analysis output results include: Use cross-entropy loss as the loss function; The empathy sentiment analysis output is obtained by applying the classifier, the loss function, and the joint multimodal representation to the output of the cross-modal fusion converter.

7. A multimodal empathy analysis system, characterized in that, The system includes: An acquisition unit is used to acquire multimodal data to be processed and extract each modal feature from the multimodal data to be processed. The feature unification unit is used to unify the features of each modality to obtain the unified features of each modality; A definition unit is used to define the unified features of each modality as first-scale language features; wherein, the first-scale language features are the base layer of the multi-scale feature pyramid; The first generation unit is used to generate semantic features of a preset scale based on the first-scale language features, wherein the preset-scale semantic features include second-scale language features and third-scale language features; the second-scale language features are used to capture sentence-level semantics; and the third-scale language features are used to capture discourse-level semantics. The second generation unit is used to generate target language features and target mixed modality features based on the adaptive mixed modality learning method, the first-scale language features, and the preset-scale semantic features; wherein, the target language features are composed of an initialized token vector, the first-scale language features, and the preset-scale semantic features, and the target mixed modality features are composed of an initialized token vector and the mixed modality features of the current layer, and the target mixed modality features are updated by updating the relationship between the language features obtained by multi-head attention computation and the remaining modality features; A cross-modal fusion unit is used to perform cross-modal fusion using the target language features and target mixed modal features to obtain a joint multimodal representation; the cross-modal fusion is a fusion operation in which the complementary relationship between mixed modalities is implicitly inferred. The empathy and sentiment analysis unit is used to perform empathy and sentiment analysis through the joint multimodal representation and obtain the empathy and sentiment analysis output results.

8. The system according to claim 7, characterized in that, The feature unification unit includes: The random initialization module is used to randomly initialize token vectors for each modality feature; The embedding module is used to embed the features of each modality into the initial token vector through a feature extractor composed of parallel fully connected layers and transformer layers, so as to obtain a unified dimension after concatenation. The input acquisition module is used to input the concatenated unified dimension into the transformer layer through the fully connected layer to obtain the unified features of each modality.

9. A storage medium, characterized in that, The storage medium includes stored instructions, wherein, when the instructions are executed, the device in which the storage medium resides is controlled to perform the multimodal empathy analysis method as described in any one of claims 1 to 6.

10. An electronic device, characterized in that, It includes a memory, and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors as described in any one of claims 1 to 6.

Images