Multi-modal data fusion method and fusion system applied to sentiment analysis

Abstract

The invention relates to the technical field of data fusion, in particular to a multi-modal data fusion method and system applied to sentiment analysis, and the method comprises the steps: obtaining multi-modal data in a target time period, and recognizing original data features corresponding to at least three types of single-modal data contained in the multi-modal data; based on a preset bidirectional long-short-term memory network and each original data feature, at least three deep semantic feature matrixes are identified from the at least three single-mode data, and each original data feature corresponds to one deep semantic feature matrix; performing feature level fusion on the at least three deep semantic feature matrixes based on a preset multi-head attention formula to obtain a primary fusion feature matrix and a secondary fusion feature matrix; and performing decision-level fusion on the primary fusion feature matrix and the secondary fusion feature matrix to obtain an emotion analysis result corresponding to the multi-modal data in the target time period. The method is convenient for improving the accuracy of the sentiment analysis result.

Description

Technical Field

[0001] The present application relates to the technical field of data fusion, and in particular to a multimodal data fusion method and fusion system for sentiment analysis. Background Art

[0002] With the popularization of social media and smart devices, people can generate and share a large amount of text, images, audio and other data anytime and anywhere to express their opinions and views. Whether it is relevant institutions, related companies or users themselves, they all hope to understand the public's emotional tendencies towards a certain event, product or service through sentiment analysis, so as to facilitate timely crisis management, brand marketing or personalized services, etc. Therefore, whether it is relevant institutions, related companies or users themselves, the demand for sentiment analysis is getting higher and higher.

[0003] In the related art, sentiment analysis of multimodal data is generally performed through a hybrid fusion method, that is, the advantages of feature-level fusion and decision-level fusion are combined to perform sentiment analysis on multimodal data. However, when fusing multimodal data based on the hybrid fusion method, the accuracy of the sentiment analysis results may be low due to the inability to capture and learn the interactive relationship between multimodal data. Summary of the invention

[0004] In order to enhance the degree of information fusion between multimodal data and thus improve the accuracy of sentiment analysis results, the present application provides a multimodal data fusion method and a fusion system for sentiment analysis.

[0005] In a first aspect, the present application provides a multimodal data fusion method for sentiment analysis, which adopts the following technical solution: A multimodal data fusion method for sentiment analysis, comprising: Acquire multimodal data within a target time period, and identify original data features corresponding to at least three types of unimodal data contained in the multimodal data; Based on a preset bidirectional long short-term memory network and each original data feature, identifying at least three deep semantic feature matrices from the at least three unimodal data, each original data feature corresponding to a deep semantic feature matrix; Based on a preset multi-head attention formula, the at least three deep semantic feature matrices are subjected to feature-level fusion to obtain a primary fusion feature matrix and a secondary fusion feature matrix; The primary fusion feature matrix and the secondary fusion feature matrix are fused at a decision level to obtain a sentiment analysis result corresponding to the multimodal data within the target time period.

[0006] By adopting the above technical solution, since the emotional state is usually affected by the context, the deep semantic feature matrix corresponding to the original data features is identified from the multimodal data through a preset bidirectional long short-term memory network, which is convenient for improving the flexibility in identifying the original data feature matrix, thereby facilitating the improvement of the accuracy in determining the deep semantic feature matrix. In addition, the deep semantic feature matrix is ​​fused at the feature level based on the preset multi-head attention formula, which is convenient for paying attention to the dependency relationship between each deep semantic feature matrix in different feature dimensions, thereby facilitating the capture of more diverse interactive information, and further facilitating the improvement of the information fusion degree between multimodal data. Finally, the primary fusion feature matrix and the secondary fusion feature matrix with a higher information fusion degree are subjected to decision-level analysis to obtain the sentiment analysis result, which is convenient for improving the accuracy of the sentiment analysis result.

[0007] In a possible implementation, the step of performing feature-level fusion on the at least three deep semantic feature matrices based on a preset multi-head attention formula to obtain a primary fused feature matrix includes: Performing feature combination on the at least three deep semantic feature matrices to obtain at least three feature groups, each feature group containing two different deep semantic feature matrices; Based on the preset multi-head attention formula and the deep semantic feature matrix contained in each feature group, determine the bimodal feature matrix corresponding to each feature group; Performing dimensionality reduction processing on each bimodal feature matrix to obtain a reduced dimensionality bimodal feature matrix corresponding to each bimodal feature matrix, determining a target deep semantic feature matrix corresponding to each reduced dimensionality bimodal feature matrix from at least three deep semantic feature matrices, and transposing each target deep semantic feature matrix to obtain a target transposed matrix corresponding to each target deep semantic feature matrix; Based on each reduced-dimensionality bimodal feature matrix and a target transposed matrix corresponding to each reduced-dimensionality bimodal feature matrix, a trimodal synergy matrix corresponding to each reduced-dimensionality bimodal feature matrix is ​​obtained; Based on each trimodal synergy matrix and the target deep semantic feature matrix corresponding to each trimodal synergy matrix, a trimodal attention matrix corresponding to each trimodal synergy matrix is ​​obtained; Based on each trimodal attention matrix and the bimodal feature matrix corresponding to each trimodal attention matrix, a trimodal collaborative attention matrix corresponding to each trimodal attention matrix is ​​obtained; All the three-modal collaborative attention matrices are concatenated to obtain the primary fusion feature matrix.

[0008] By adopting the above technical solution, the primary fusion feature matrix obtained through multiple splicing, matrix multiplication and other operations is convenient for more comprehensive capture of the interactive information and mutual relationships between different single-modal data. By introducing the trimodal collaborative attention matrix, it is convenient to focus on strengthening the relevant content in the multimodal data and weakening the irrelevant content in the multimodal data, thereby facilitating the improvement of the interactivity between the multimodal data, and then facilitating the improvement of the accuracy of the sentiment analysis results.

[0009] In a possible implementation, when the deep semantic feature matrix included in the feature group is a text deep semantic feature matrix and a visual deep semantic feature matrix, based on a preset multi-head attention formula and the deep semantic feature matrix included in the feature group, determining a bimodal feature matrix corresponding to the feature group includes: Obtaining a preset trainable weight matrix, and determining a projection feature matrix based on the preset trainable weight matrix and a deep semantic feature matrix included in the feature group, wherein the preset trainable weight matrix includes a query trainable weight matrix, a key trainable weight matrix, and a value trainable weight matrix, The projection feature matrix includes a query key matrix corresponding to the text depth semantic feature matrix, a key key matrix and a value key matrix corresponding to the visual depth semantic feature matrix; Based on the preset multi-head attention formula, the query key matrix corresponding to the text deep semantic feature matrix, and the key key matrix and value key matrix corresponding to the visual deep semantic feature matrix, multiple attention scores corresponding to the feature group are determined. The preset multi-head attention formula is: , where Attention i is the i-th attention score corresponding to the feature group, i is a positive integer, and is the normalization factor, d k is the vector dimension of the key matrix corresponding to the visual depth semantic feature matrix; All attention scores corresponding to the feature group are concatenated to obtain a multi-head output result of the feature group, and a bimodal feature matrix corresponding to the feature group is determined based on the multi-head output result and a preset trainable linear transformation matrix.

[0010] By adopting the above technical solution, the attention score between two deep semantic feature matrices is evaluated on different feature dimensions by presetting the multi-head attention formula, thereby facilitating the improvement of the interactivity between different deep semantic feature matrices, that is, facilitating the improvement of the interactivity between different unimodal data, and further facilitating the enhancement of the information fusion between unimodal data.

[0011] In a possible implementation, when the deep semantic feature matrix included in the feature group is a text deep semantic feature matrix and a visual deep semantic feature matrix, determining the projection feature matrix based on the preset trainable weight matrix and the deep semantic feature matrix included in the feature group includes: Determine the query key matrix corresponding to the text deep semantic feature matrix according to the text deep semantic feature matrix, the query trainable weight matrix and the first calculation formula, wherein the first calculation formula is: , Q T is the query key matrix corresponding to the text deep semantic feature matrix, T is the text deep semantic feature matrix, W i Q To query the trainable weight matrix; According to the visual depth semantic feature matrix, the key trainable weight matrix and the second calculation formula, the key key matrix corresponding to the visual depth semantic feature matrix is ​​determined, and the second calculation formula is: , K V is the key matrix corresponding to the visual depth semantic feature matrix, V is the visual depth semantic feature matrix, W i K is the key trainable weight matrix; According to the visual depth semantic feature matrix, the value trainable weight matrix and the third calculation formula, the value key matrix corresponding to the visual depth semantic feature matrix is ​​determined, and the third calculation formula is: , V V is the value key matrix corresponding to the visual depth semantic feature matrix, V is the visual depth semantic feature matrix, W i V is the value trainable weight matrix; The query key matrix corresponding to the text depth semantic feature matrix, the key key matrix corresponding to the visual depth semantic feature matrix, and the value key matrix constitute a projection feature matrix.

[0012] By adopting the above technical solution, the present application provides a method for determining a projection feature matrix.

[0013] In a possible implementation, the at least three deep semantic feature matrices are subjected to feature-level fusion based on a preset multi-head attention formula to obtain a secondary fusion feature matrix, including: Concatenating at least three bimodal feature matrices to obtain a concatenated trimodal feature matrix; Performing dimensionality reduction processing on the spliced ​​trimodal feature matrix to obtain a reduced-dimensional spliced ​​trimodal feature matrix, and determining the self-attention distribution of the reduced-dimensional spliced ​​trimodal feature matrix based on a preset self-attention mechanism to obtain a self-attention spliced ​​trimodal feature matrix; The self-attention spliced ​​trimodal feature matrix and the at least three deep semantic feature matrices are spliced ​​to obtain a secondary fusion feature matrix.

[0014] By adopting the above technical solution, by splicing the bimodal feature matrix and performing feature fusion based on the spliced ​​trimodal feature matrix, it is convenient to improve the comprehensiveness of the feature fusion process. In addition, the attention distribution of the spliced ​​trimodal feature matrix is ​​determined based on the preset multi-head attention mechanism, which facilitates a more comprehensive understanding of the deep-level interaction between the trimodal feature matrices, thereby facilitating improving the accuracy of determining the secondary content and feature matrix.

[0015] In a possible implementation, the performing decision-level fusion on the primary fusion feature matrix and the secondary fusion feature matrix to obtain a sentiment analysis result corresponding to the multimodal data within the target time period includes: Based on a preset classifier, the primary fusion feature matrix and the secondary fusion feature matrix are respectively classified to determine a target classification probability score; Based on the target classification probability score and the preset emotion category mapping relationship, determine the target emotion category corresponding to the target classification probability, wherein the preset emotion category mapping relationship is the corresponding relationship between the target classification probability and the target emotion category; Generate a sentiment analysis result based on the target sentiment category.

[0016] By adopting the above technical solution, after obtaining the primary fusion feature matrix and the secondary fusion feature matrix through the feature level fusion operation, the primary fusion feature matrix and the secondary fusion feature matrix are fused at the decision level, so as to improve the accuracy and robustness of sentiment analysis.

[0017] In a possible implementation, the classifying the primary fusion feature matrix and the secondary fusion feature matrix based on a preset classifier to determine a target classification probability score includes: Obtaining classifier weight information and classifier bias information corresponding to the classifier, the classifier weight information including a first classifier weight and a second classifier weight, and the classifier bias information including a first classifier bias and a second classifier bias; The primary classification probability score corresponding to the primary fusion feature matrix is ​​determined based on the first classifier weight, the first classifier bias, the primary fusion feature matrix and a first classification formula, wherein the first classification formula is: , where l1 is the primary classification probability score, W1 is the first classifier weight, b1 is the first classifier bias, and P is the primary fusion feature matrix; The secondary classification probability score corresponding to the secondary fusion feature matrix is ​​determined based on the second classifier weight, the second classifier bias, the secondary fusion feature matrix and a second classification formula, wherein the second classification formula is: , where l2 is the secondary classification probability score, W2 is the second classifier weight, b2 is the second classifier bias, and S is the secondary fusion feature matrix; The target classification probability score is determined from the primary classification probability score and the secondary classification probability score based on a preset rule.

[0018] By adopting the above technical solution, the present application provides a method for determining a target classification probability score.

[0019] In a second aspect, the present application provides a fusion system, which adopts the following technical solution: A fusion system, the fusion system comprising: at least one processor; Memory; At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the above-mentioned multimodal data fusion method applied to sentiment analysis.

[0020] In a third aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium includes: a computer program that can be loaded by a processor and execute the multimodal data fusion method applied to sentiment analysis.

[0021] In a fourth aspect, the present application provides a computer program product, which adopts the following technical solution: A computer program product includes a computer program, and when the computer program is executed by a processor, the multimodal data fusion method applied to sentiment analysis is implemented.

[0022] In summary, the present application includes at least one of the following beneficial technical effects: Since emotional states are usually affected by context, a deep semantic feature matrix corresponding to the original data features is identified from multimodal data by presetting a bidirectional long short-term memory network, so as to improve the flexibility in identifying the original data feature matrix, thereby improving the accuracy in determining the deep semantic feature matrix. In addition, the deep semantic feature matrix is ​​fused at the feature level based on a preset multi-head attention formula, so as to focus on the dependency relationship between each deep semantic feature matrix in different feature dimensions, thereby facilitating the capture of more diverse interactive information, and further facilitating the improvement of the information fusion degree between multimodal data. Finally, the primary fusion feature matrix and the secondary fusion feature matrix with a high degree of information fusion are subjected to decision-level analysis to obtain the sentiment analysis result, so as to improve the accuracy of the sentiment analysis result. The primary fusion feature matrix obtained through multiple splicing, matrix multiplication and other operations makes it easier to capture the interactive information and relationships between different single-modal data more comprehensively. By introducing the trimodal collaborative attention matrix, it is easier to focus on strengthening the relevant content in the multimodal data and weaken the irrelevant content in the multimodal data, thereby facilitating the improvement of the interactivity between multimodal data and the accuracy of the sentiment analysis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flowchart of a multimodal data fusion method applied to sentiment analysis in an embodiment of the present application; Figure 2 It is a schematic diagram of a process for determining a primary fusion feature matrix in an embodiment of the present application; Figure 3 is a schematic diagram of a process for determining a secondary fusion feature matrix in an embodiment of the present application; Figure 4 It is a structural diagram of a fusion system in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following is combined with Figures 1 to 4 This application is described in further detail.

[0025] After reading this specification, those skilled in the art may make non-creative modifications to this embodiment as needed, but such modifications are protected by patent law as long as they are within the scope of the claims of this application.

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] It should be noted that in the optional embodiments of the present application, the object information and other related data involved, when the embodiments in the present application are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the embodiments of the present application involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information needs to obtain the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained. The embodiments also need to be implemented with the authorization and consent of the object.

[0028] Specifically, the embodiment of the present application provides a multimodal data fusion method for sentiment analysis, which is executed by a fusion system, and the fusion system can be a server or a terminal device, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and the embodiment of the present application does not limit this.

[0029] refer to Figure 1 , Figure 1 : is a flow chart of a multimodal data fusion method for sentiment analysis in an embodiment of the present application, the method comprising steps S110 to S140, wherein: Step S110: Acquire multimodal data within a target time period, and identify original data features corresponding to at least three types of unimodal data contained in the multimodal data.

[0030] Specifically, the duration corresponding to the target time period can be 30 minutes or 3 hours. The specific duration is not specifically limited in the embodiment of the present application, as long as it can ensure that the three unimodal data contained in the multimodal data are synchronized data. At least three unimodal data in the multimodal data can be voice data, text data and visual data, and the visual data can be data such as images or videos. In order to improve the accuracy of sentiment analysis based on multimodal data, at least three unimodal data in the multimodal data need to be fused in the embodiment of the present application, where voice data contains emotional clues such as intonation, speech speed, and volume, which helps to perceive the user's emotions; text data provides rich semantic information, which helps to understand the user's intuitive feelings; visual data can capture the user's facial expressions, eye contact, etc., which helps to understand the user's visual emotions. Therefore, the three unimodal data in the embodiment of the present application are voice data, text data, and visual data. In addition, the user's physiological data can also be analyzed, such as heart rate, blood pressure, etc. The specific unimodal data type is not specifically limited in the embodiment of the present application, and can be selected or set according to the actual needs of relevant technical personnel.

[0031] The original data features corresponding to the unimodal data are used to characterize the basic attributes or features of the corresponding unimodal data. They are intuitive and easy to understand, and may contain a large amount of redundant information, that is, for sentiment analysis, not all original data features are valid data features. Convolutional neural networks, 3D-CNN networks, and OpenSMILE toolkits can be used to extract original speech features, original text features, and original visual features, respectively, where the embedding dimensions of the original speech features, original text features, and original visual features can be set to 100, 100, and 73, respectively. The specific preset feature recognition algorithm is not specifically limited in the embodiments of this application, and can be a SIFT algorithm, a SURF algorithm, etc., as long as the corresponding original data features can be identified from the unimodal data.

[0032] Step S120: Based on a preset bidirectional long short-term memory network and each original data feature, at least three deep semantic feature matrices are identified from at least three unimodal data, and each original data feature corresponds to a deep semantic feature matrix.

[0033] Specifically, based on the preset bidirectional long short-term memory network, the deep semantic feature matrix corresponding to each original data feature can be determined, that is, the deep semantic feature matrix corresponding to each unimodal data can be determined. Among them, the preset bidirectional long short-term memory network (Bidirectional Long Short-Term Memory, BiLSTM) contains two directional LSTMs, namely forward LSTM and backward LSTM. The bidirectional LSTM can process forward and backward sentiment information at the same time, ensuring that each original data feature can be well explained in the context of unimodal data, that is, it is convenient to select effective data features more flexibly. The speech unimodal data can determine the speech depth semantic matrix through the first BiLSTM and the original speech features, the text unimodal data can determine the text depth semantic matrix through the second BiLSTM and the original text features, and the visual unimodal data can determine the visual depth semantic matrix through the third BiLSTM and the original visual features. Among them, the first BiLSTM network, the second BiLSTM network and the third BiLSTM network all refer to BiLSTM network models obtained after training with their respective unimodal data. When training the BiLSTM network model, 300 or 200 neurons can be selected. The specific number of neurons is not specifically limited in the embodiment of the present application and can be set by relevant technical personnel according to actual needs.

[0034] In order to facilitate the subsequent analysis or processing of at least three deep semantic feature matrices, after determining the at least three deep semantic feature matrices, the at least three deep semantic features can be passed into the fully connected layer, and then the at least three deep semantic feature matrices can be mapped to the same dimension through linear transformation and non-linear activation function. The specific dimensionality reduction method is not specifically limited in the embodiments of the present application, as long as the at least three deep semantic feature matrices can be mapped to the same dimension.

[0035] Step S130: Based on a preset multi-head attention formula, at least three deep semantic feature matrices are subjected to feature-level fusion to obtain a primary fusion feature matrix and a secondary fusion feature matrix.

[0036] Specifically, before obtaining the primary fusion feature matrix, it is necessary to first perform pairwise fusion of at least three deep semantic feature matrices, and then perform attention distribution calculation on the pairwise fusion results based on the preset multi-head attention formula to obtain at least three bimodal feature matrices, and then perform matrix concatenation or multiplication of at least three bimodal feature matrices to generate a primary trimodal feature matrix, and finally perform probability distribution calculation on the primary trimodal feature matrix to obtain the primary fusion feature matrix.

[0037] Before obtaining the secondary fusion feature matrix, it is necessary to concatenate the pairwise fusion results of the three deep semantic feature matrices to obtain the secondary trimodal feature matrix. Then, it is necessary to calculate the attention distribution of the secondary trimodal feature matrix according to the preset multi-head attention formula. Finally, the secondary fusion feature matrix is ​​determined based on the secondary trimodal feature matrix with attention information.

[0038] Among them, by presetting the multi-head attention formula, the attention scores of different feature dimensions can be calculated in parallel. That is, by presetting the multi-head attention formula, different relationships between different unimodal data can be paid attention to in different feature dimensions, thereby capturing more diverse interaction information. Compared with the single attention mechanism, it is easier to mine the various dependencies and interaction information hidden between different unimodal data.

[0039] Step S140: Perform decision-level fusion on the primary fusion feature matrix and the secondary fusion feature matrix to obtain sentiment analysis results corresponding to the multimodal data within the target time period.

[0040] Specifically, decision-level fusion is a higher-level fusion method that generates the final judgment, i.e., the sentiment analysis result, by combining and determining the primary fusion feature matrix and the secondary fusion feature matrix. The advantage of this method is that it has strong anti-interference and fault tolerance capabilities. Since the classification errors of different unimodal data usually do not affect each other, that is, the accuracy of the overall sentiment analysis result will not be significantly reduced due to the error of a certain unimodal data, thereby avoiding the superposition of errors. Decision-level fusion can be achieved by using result weighting, product operation, maximum or minimum value selection, median calculation, etc., which are not specifically limited in the embodiments of this application.

[0041] Furthermore, when the primary fusion feature matrix and the secondary fusion feature matrix are fused at the decision level to obtain the sentiment analysis result corresponding to the multimodal data in the target time period, the following may be specifically included: Based on the preset classifier, the primary fusion feature matrix and the secondary fusion feature matrix are classified respectively to determine the target classification probability score; based on the target classification probability score and the preset emotion category mapping relationship, the target emotion category corresponding to the target classification probability is determined, and the preset emotion category mapping relationship is the correspondence between the target classification probability and the target emotion category; based on the target emotion category, the emotion analysis result is generated.

[0042] Specifically, the preset classifier can be a softmax classifier or other classifiers. The specific classifier is not specifically limited in the embodiment of the present application. As long as the primary classification probability score corresponding to the primary fusion feature matrix and the secondary classification probability score corresponding to the secondary fusion feature matrix can be determined, the target classification probability score may be the maximum value, minimum value or mean value between the primary classification probability score and the secondary classification probability score. The specific determination method can be set by relevant technical personnel. Among them, the embodiment of the present application provides a method for determining the target classification probability score, which may specifically include: Obtain classifier weight information and classifier bias information corresponding to the classifier, the classifier weight information includes a first classifier weight and a second classifier weight, and the classifier bias information includes a first classifier bias and a second classifier bias; determine the primary classification probability score corresponding to the primary fusion feature matrix based on the first classifier weight, the first classifier bias, the primary fusion feature matrix, and the first classification formula, the first classification formula is , where l1 is the primary classification probability score, W1 is the first classifier weight, b1 is the first classifier bias, and P is the primary fusion feature matrix; the secondary classification probability score corresponding to the secondary fusion feature matrix is ​​determined based on the second classifier weight, the second classifier bias, the secondary fusion feature matrix and the second classification formula. The second classification formula is , where l2 is the secondary classification probability score, W2 is the second classifier weight, b2 is the second classifier bias, and S is the secondary fusion feature matrix; the target classification probability score is determined from the primary classification probability score and the secondary classification probability score based on preset rules.

[0043] Specifically, the classifier weight information and classifier bias information corresponding to the classifier can be uploaded to the fusion system in advance by relevant staff. The specific first classifier weight and the second classifier weight, as well as the specific first classifier bias and the second classifier bias are not specifically limited in the embodiment of the present application. The primary fusion feature matrix and the secondary fusion feature matrix can be separately classified based on their respective corresponding classification formulas to obtain their respective corresponding primary classification probability scores and secondary classification probability scores, and finally the preset rule, i.e., the Maximum rule, is used to determine the target classification probability scores corresponding to the primary classification probability scores and the secondary classification probability scores, where:

[0044] Among them, l1 is the primary classification probability score, W1 is the first classifier weight, b1 is the first classifier bias, l2 is the secondary classification probability score, W2 is the second classifier weight, b2 is the second classifier bias, P is the primary fusion feature matrix, and S is the secondary fusion feature matrix.

[0045] Different target classification probabilities correspond to different target emotion categories. Based on the preset emotion category mapping relationship, the target emotion category corresponding to any target classification probability can be determined. The specific content of the preset emotion category mapping relationship is not specifically limited in the embodiment of this application, and can be determined by relevant staff based on historical experimental data and uploaded to the fusion system. The emotion analysis results generated based on the target emotion category facilitate the intuitiveness of the feedback process, that is, by viewing the emotion analysis results, the user's emotional state in the target time period can be directly determined.

[0046] For the embodiments of the present application, since the emotional state is usually affected by the context, the deep semantic feature matrix corresponding to the original data features is identified from the multimodal data by presetting a bidirectional long short-term memory network, so as to improve the flexibility in identifying the original data feature matrix, thereby improving the accuracy in determining the deep semantic feature matrix. In addition, the deep semantic feature matrix is ​​fused at the feature level based on the preset multi-head attention formula, so as to pay attention to the dependency relationship between each deep semantic feature matrix in different feature dimensions, so as to capture more diverse interactive information, and then to improve the information fusion degree between multimodal data. Finally, the primary fusion feature matrix and the secondary fusion feature matrix with a higher information fusion degree are subjected to decision-level analysis to obtain the sentiment analysis result, so as to improve the accuracy of the sentiment analysis result.

[0047] Furthermore, the specific process of performing feature-level fusion on at least three deep semantic feature matrices based on the preset multi-head attention formula to obtain a primary fusion feature matrix may include steps Sa1 to Sa7, such as Figure 2 As shown, where: Step Sa1: Perform feature combination on at least three deep semantic feature matrices to obtain at least three feature groups, each feature group containing two different deep semantic feature matrices.

[0048] Specifically, at least three deep semantic feature matrices are combined in pairs to obtain at least three feature groups. For example, the three existing deep semantic feature matrices are a speech deep semantic feature matrix, a text deep semantic feature matrix, and a visual deep semantic feature matrix. The speech deep semantic feature matrix and the text deep semantic feature matrix can be combined to obtain feature group a, the speech deep semantic feature matrix and the visual deep semantic feature matrix can be combined to obtain feature group b, and the text deep semantic feature matrix and the visual deep semantic feature matrix can be combined to obtain feature group c.

[0049] Step Sa2: Based on the preset multi-head attention formula and the deep semantic feature matrix contained in each feature group, determine the bimodal feature matrix corresponding to each feature group.

[0050] Specifically, by presetting the multi-head attention formula, the features of different unimodal data belonging to the feature group are fused, and different attention heads will calculate the attention weights corresponding to the feature group on different feature dimensions, thereby enhancing the accuracy of the cross-modal data fusion process. The specific preset multi-head attention formula is not specifically limited in the embodiments of the present application, as long as the attention result corresponding to the feature group can be determined, and finally the bimodal feature matrix corresponding to the feature group is determined based on the attention result corresponding to the feature group.

[0051] Further, when the deep semantic feature matrix included in the feature group is a text deep semantic feature matrix and a visual deep semantic feature matrix, based on a preset multi-head attention formula and the deep semantic feature matrix included in the feature group, determining the bimodal feature matrix corresponding to the feature group may specifically include: Obtain a preset trainable weight matrix, and determine a projection feature matrix based on the preset trainable weight matrix and the deep semantic feature matrix contained in the feature group, the preset trainable weight matrix includes a query trainable weight matrix, a key trainable weight matrix, and a value trainable weight matrix, and the projection feature matrix includes a query key matrix corresponding to the text deep semantic feature matrix, a key key matrix and a value key matrix corresponding to the visual deep semantic feature matrix.

[0052] Specifically, the preset trainable weight matrix can be determined by relevant staff based on historical experimental data and uploaded to the fusion system. The preset trainable weight matrix includes the query trainable weight matrix W i Q , key trainable weight matrix W i K and the value of the trainable weight matrix W i V The projection feature matrix contains the query key matrix Q corresponding to the text deep semantic feature matrix T , the key matrix K corresponding to the visual depth semantic feature matrix V Sum value key matrix V V , where Q T is the query matrix extracted from the text data, which is used to find the correlation between the text deep semantic feature matrix in the text data and other modal data; K V V is the key matrix extracted from the visual data, which is used to match the query key matrix to identify which visual deep semantic feature matrices are related to the query in the text data from the visual data; V is a matrix of values ​​extracted from the visual data that represents what is actually returned by the attention mechanism.

[0053] In an embodiment of the present application, a method for determining a projection feature matrix is ​​provided, which may specifically include: According to the text deep semantic feature matrix, the query trainable weight matrix and the first calculation formula, the query key matrix corresponding to the text deep semantic feature matrix is ​​determined, wherein the first calculation formula is: , Q T is the query key matrix corresponding to the text deep semantic feature matrix, T is the text deep semantic feature matrix, W i Q To query the trainable weight matrix; according to the visual depth semantic feature matrix, the key trainable weight matrix and the second calculation formula, determine the key key matrix corresponding to the visual depth semantic feature matrix, the second calculation formula is , K V is the key matrix corresponding to the visual depth semantic feature matrix, V is the visual depth semantic feature matrix, W i K is the key trainable weight matrix; according to the visual depth semantic feature matrix, the value trainable weight matrix and the third calculation formula, the value key matrix corresponding to the visual depth semantic feature matrix is ​​determined, and the third calculation formula is , V V is the value key matrix corresponding to the visual depth semantic feature matrix, V is the visual depth semantic feature matrix, W i V is a value trainable weight matrix; the query key matrix corresponding to the text deep semantic feature matrix, the key key matrix corresponding to the visual deep semantic feature matrix, and the value key matrix constitute the projection feature matrix.

[0054] Specifically, the first calculation formula, the second calculation formula, and the third calculation formula are summarized, and the summary result is as follows:

[0055] Among them, T is the text deep semantic feature matrix, and V is the visual deep semantic feature matrix, which contains the deep semantic information corresponding to the text and visual data respectively. i Q To query the trainable weight matrix, W i K is the key trainable weight matrix, W i V The trainable weight matrix can be obtained or identified by relevant staff after uploading it to the fusion system in advance. The trainable weight matrix is ​​a set of parameters used to encode the connection strength between neurons in each layer in the neural network, which facilitates better fitting of different single-modal data in the information fusion process. Represents the multiplication operation between matrices. Determine Q T , K V and V V Then, the projection feature matrix can be obtained.

[0056] Based on the preset multi-head attention formula, the query key matrix corresponding to the text deep semantic feature matrix, and the key key matrix and value key matrix corresponding to the visual deep semantic feature matrix, multiple attention scores corresponding to the feature group are determined. The preset multi-head attention formula is: in, is the i-th attention score corresponding to the feature group, i is a positive integer is the normalization factor, is the vector dimension of the key matrix corresponding to the visual depth semantic feature matrix.

[0057] Specifically, the vector dimensions in the key matrix can be identified to determine , and then The normalization factor corresponding to the vector dimension is obtained by performing a root calculation. When determining the attention result corresponding to the feature group, the normalization factor is added to prevent the problem of excessive value caused by the increase of the feature vector dimension, thereby helping to improve the stability of the attention result. V Perform the transpose operation to obtain , and then the Q determined in the above embodiment T 、V V as well as Imported into the preset multi-head attention formula, the obtained Q T 、V V as well as May be different, therefore, multiple attention scores corresponding to the feature groups can be obtained after multiple calculations.

[0058] All attention scores corresponding to the feature group are concatenated to obtain the multi-head output result of the feature group, and the bimodal feature matrix corresponding to the feature group is determined based on the multi-head output result and the preset trainable linear transformation matrix.

[0059] Specifically, all attention scores can be imported into the attention score concatenation formula to obtain multi-head output results, where the attention score concatenation formula can be: MHO=Concat(Attention1, Attention2,…,Attentionn); Among them, MHO is used to characterize the multi-head output results, and Concat is used to characterize the splicing operation. After obtaining the multi-head output results, the multi-head output results are multiplied with the preset trainable linear transformation matrix to obtain the bimodal feature matrix corresponding to the feature group. Among them, the preset trainable linear transformation matrix is ​​used to perform linear transformation on the spliced ​​multi-head output results in the multi-head attention mechanism.

[0060] Since the feature group contains the text depth semantic feature matrix and the visual depth semantic feature matrix, the obtained bimodal feature matrix, that is, the text-visual bimodal feature matrix can be expressed as DF TV Based on the above method, two other bimodal feature matrices can be obtained. That is, when the feature group includes the speech deep semantic feature matrix and the text deep semantic feature matrix, the corresponding bimodal feature matrix is ​​the speech-text bimodal feature matrix, which can be expressed by DF AT Indicates that DF can also be used TA ; When the feature group contains the speech depth semantic feature matrix and the visual depth semantic feature matrix, the corresponding bimodal feature matrix is ​​the speech-visual bimodal feature matrix, which can be expressed as DF AV express.

[0061] Step Sa3: Perform dimensionality reduction processing on each bimodal feature matrix to obtain a reduced dimensionality bimodal feature matrix corresponding to each bimodal feature matrix, determine the target deep semantic feature matrix corresponding to each reduced dimensionality bimodal feature matrix from at least three deep semantic feature matrices, and transpose each target deep semantic feature matrix to obtain a target transposed matrix corresponding to each target deep semantic feature matrix.

[0062] Step Sa4: Based on each reduced-dimensionality bimodal feature matrix and the target transposed matrix corresponding to each reduced-dimensionality bimodal feature matrix, a trimodal synergy matrix corresponding to each reduced-dimensionality bimodal feature matrix is ​​obtained.

[0063] Specifically, since the bimodal feature matrix often contains a large amount of feature information and has a higher feature dimension, directly splicing or fusing the bimodal feature matrix with a higher feature dimension with the deep semantic feature matrix may lead to a sharp increase in the amount of calculation and may also affect the computing efficiency. Therefore, when the bimodal feature matrix needs to be fused with other deep semantic feature matrices, it is necessary to first perform dimensionality reduction processing on the bimodal feature matrix to obtain a reduced dimensionality bimodal feature matrix of the bimodal feature matrix.

[0064] The bimodal feature matrix that needs to be reduced in dimension can be imported into the fully connected layer for dimensionality reduction. The bimodal feature matrix after dimensionality reduction needs to be in the same feature dimension as the deep semantic feature matrix corresponding to the unimodal data. VT For example, after dimensionality reduction processing, the reduced-dimensional bimodal feature matrix can be obtained using F VT Indicates that the dimension reduction formula can be ; Among them, Dense represents the fully connected layer, F VTThe corresponding feature dimensions have the same feature dimensions as the speech deep semantic feature matrix A, the text deep semantic feature matrix T, and the visual deep semantic feature matrix A. Different reduced-dimensional bimodal feature matrices correspond to different target deep semantic feature matrices. The target deep semantic feature matrix corresponding to the reduced-dimensional bimodal feature matrix is ​​the deep semantic feature matrix not included in the reduced-dimensional bimodal feature matrix. For example, when the reduced-dimensional bimodal feature matrix is ​​F VT When , the corresponding target deep semantic feature matrix is ​​A; when the dimension-reduced bimodal feature matrix is ​​F AT When , the corresponding target deep semantic feature matrix is ​​V; when the dimension-reduced bimodal feature matrix is ​​F AV When , the corresponding target deep semantic feature matrix is ​​T.

[0065] After determining the target deep semantic feature matrix corresponding to each dimension-reduced bimodal feature matrix, it is necessary to perform a transposition operation on the target deep semantic feature matrix. For example, the target deep semantic feature matrix A can be expressed as A after the transposition operation. T .

[0066] The reduced-dimension bimodal feature matrix and the corresponding target transposed matrix are subjected to matrix multiplication operation to obtain the trimodal synergy matrix corresponding to the reduced-dimension bimodal feature matrix. Specifically, the reduced-dimension bimodal feature matrix and the corresponding target transposed matrix can be introduced into the trimodal synergy formula for matrix multiplication to obtain the reduced-dimension bimodal feature matrix F VT For example, the dimension-reduced bimodal feature matrix F VT The corresponding target transposed matrix is ​​F VT , , where F VTA F VT Based on the above method, the trimodal synergy matrix corresponding to each dimension-reduced bimodal feature matrix can be determined, for example, F ATV F AT The corresponding three-modal coordination matrix, F AVT F AV The corresponding three-modal synergy matrix.

[0067] Step Sa5: Based on each trimodal synergy matrix and the target deep semantic feature matrix corresponding to each trimodal synergy matrix, obtain the trimodal attention matrix corresponding to each trimodal synergy matrix.

[0068] Specifically, before determining the trimodal attention matrix corresponding to the trimodal synergy matrix, it is necessary to calculate the probability distribution score corresponding to the trimodal synergy matrix based on the Softmax function to obtain the trimodal probability distribution matrix, and then perform matrix multiplication on the trimodal probability distribution matrix and the corresponding target deep semantic features, where the trimodal synergy matrix F VTA For example, ,in is the three-modal coordination matrix F VTA The corresponding trimodal probability distribution matrix is ​​obtained by multiplying the trimodal synergy matrix with the corresponding target deep semantic feature matrix to obtain the trimodal attention matrix corresponding to the trimodal synergy matrix. For example, given F VTA The corresponding target deep attention matrix is ​​the speech deep semantic feature matrix A. Therefore, the trimodal attention matrix corresponding to the trimodal probability distribution matrix is Based on the above method, the trimodal attention matrix corresponding to each trimodal synergy matrix can be determined. For example, the trimodal synergy matrix F ATV The corresponding trimodal attention matrix is ​​P ATV , the three-modal coordination matrix F AVT The corresponding trimodal attention matrix is ​​P AVT .

[0069] Step Sa6: Based on each trimodal attention matrix and the bimodal feature matrix corresponding to each trimodal attention matrix, a trimodal collaborative attention matrix corresponding to each trimodal attention matrix is ​​obtained.

[0070] Specifically, the trimodal attention matrix and the corresponding bimodal feature matrix are subjected to Hadamard product operation to obtain the trimodal collaborative attention matrix corresponding to the trimodal attention matrix. VTA For example, given P VTA The corresponding bimodal feature matrix is ​​F VT , P VTA and F VT By importing the collaborative attention formula, we can get the corresponding trimodal collaborative attention matrix, where the collaborative attention formula is: ,in, Represents the Hadamard product operation. Based on the above method, the trimodal collaborative attention matrix corresponding to each trimodal attention matrix can be determined. For example, the trimodal attention matrix P ATV The corresponding trimodal collaborative attention matrix is , the trimodal attention matrix P AVT The corresponding trimodal collaborative attention matrix is .

[0071] Step Sa7: Concatenate all the three-modal collaborative attention matrices to obtain the primary fusion feature matrix.

[0072] Specifically, all the three-modal collaborative attention matrices are concatenated as follows: , where P is the primary fusion feature matrix.

[0073] The primary fusion feature matrix obtained through multiple splicing, matrix multiplication and other operations makes it easier to capture the interactive information and relationships between different single-modal data more comprehensively. By introducing the trimodal collaborative attention matrix, it is easier to focus on strengthening the relevant content in the multimodal data and weaken the irrelevant content in the multimodal data, thereby facilitating the improvement of the interactivity between multimodal data and the accuracy of the sentiment analysis results.

[0074] Further, based on the preset multi-head attention formula, at least three deep semantic feature matrices are subjected to feature-level fusion to obtain a secondary fusion feature matrix, which may specifically include steps Sb1 to Sb3, such as Figure 3 As shown, where: Step Sb1: concatenate at least three bimodal feature matrices to obtain a concatenated trimodal feature matrix.

[0075] Specifically, the at least three determined bimodal feature matrices are directly concatenated, that is, the text-visual bimodal feature matrix DF TV , speech-text bimodal feature matrix DF AT And the speech-vision bimodal feature matrix, available DF AV After splicing, the obtained spliced ​​three-modal feature matrix can be used AVT Indicates that you can also use B ATV or B TVA express.

[0076] Step Sb2: Perform dimensionality reduction processing on the spliced ​​trimodal feature matrix to obtain a reduced-dimensional spliced ​​trimodal feature matrix, and determine the self-attention distribution of the reduced-dimensional spliced ​​trimodal feature matrix based on a preset self-attention mechanism to obtain a self-attention spliced ​​trimodal feature matrix.

[0077] Specifically, the concatenated three-modal feature matrix can be imported into the fully connected layer for dimensionality reduction, such as: , where C AVT For B AVT The dimensionality reduction and concatenation of the three-modal feature matrix after dimensionality reduction.

[0078] Then, the dimensionality reduction and splicing trimodal feature matrix is ​​imported into the self-attention distribution calculation formula to perform self-attention distribution calculation, and the attention splicing trimodal feature matrix corresponding to the dimensionality reduction and splicing trimodal feature matrix is ​​obtained, such as: , where S AVT C AVT After self-attention distribution processing, the self-attention splicing trimodal feature matrix is ​​obtained, and SelfAttention represents the preset self-attention mechanism.

[0079] Step Sb3: Concatenate the self-attention trimodal feature matrix and at least three deep semantic feature matrices to obtain a secondary fusion feature matrix.

[0080] Specifically, all deep semantic feature matrices are concatenated with the self-attention concatenated trimodal feature matrix with attention information to obtain a secondary fusion feature matrix, such as: .

[0081] By splicing the bimodal feature matrix and performing feature fusion based on the spliced ​​trimodal feature matrix, the comprehensiveness of the feature fusion process can be improved. In addition, the attention distribution of the spliced ​​trimodal feature matrix is ​​determined based on the preset multi-head attention mechanism, which facilitates a more comprehensive understanding of the deep-level interaction between the trimodal feature matrices, thereby facilitating the improvement of the accuracy of determining the secondary content and feature matrix.

[0082] The present application embodiment provides a fusion system, such as Figure 4 As shown, Figure 4 The fusion system 400 shown includes: a processor 401 and a memory 403. The processor 401 and the memory 403 are connected, such as through a bus 402. Optionally, the fusion system 400 may also include a transceiver 404. It should be noted that in actual applications, the transceiver 304 is not limited to one, and the structure of the fusion system 400 does not constitute a limitation on the embodiments of the present application.

[0083] Processor 401 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 401 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0084] The bus 402 may include a path to transmit information between the above components. The bus 402 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 402 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The fact that only one line is used in the diagram does not mean that there is only one bus or only one type of bus.

[0085] The memory 403 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0086] The memory 403 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 401. The processor 401 is used to execute the application code stored in the memory 403 to implement the contents shown in the above method embodiment.

[0087] The converged system includes, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs, desktop computers, etc. It can also be a server, etc. Figure 4 The fusion system shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0088] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0089] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method in any of the above embodiments is implemented.

[0090] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0091] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A multimodal data fusion method for sentiment analysis, characterized in that: include: Acquire multimodal data within a target time period, and identify original data features corresponding to at least three types of unimodal data contained in the multimodal data; Based on a preset bidirectional long short-term memory network and each original data feature, identifying at least three deep semantic feature matrices from the at least three unimodal data, each original data feature corresponding to a deep semantic feature matrix; Based on a preset multi-head attention formula, the at least three deep semantic feature matrices are subjected to feature-level fusion to obtain a primary fusion feature matrix and a secondary fusion feature matrix; The primary fusion feature matrix and the secondary fusion feature matrix are fused at a decision level to obtain a sentiment analysis result corresponding to the multimodal data within the target time period.

2. The multimodal data fusion method for sentiment analysis according to claim 1, characterized in that: The step of performing feature-level fusion on the at least three deep semantic feature matrices based on a preset multi-head attention formula to obtain a primary fusion feature matrix includes: Performing feature combination on the at least three deep semantic feature matrices to obtain at least three feature groups, each feature group containing two different deep semantic feature matrices; Based on the preset multi-head attention formula and the deep semantic feature matrix contained in each feature group, determine the bimodal feature matrix corresponding to each feature group; Performing dimensionality reduction processing on each bimodal feature matrix to obtain a reduced dimensionality bimodal feature matrix corresponding to each bimodal feature matrix, determining a target deep semantic feature matrix corresponding to each reduced dimensionality bimodal feature matrix from at least three deep semantic feature matrices, and transposing each target deep semantic feature matrix to obtain a target transposed matrix corresponding to each target deep semantic feature matrix; Based on each reduced-dimensionality bimodal feature matrix and a target transposed matrix corresponding to each reduced-dimensionality bimodal feature matrix, a trimodal synergy matrix corresponding to each reduced-dimensionality bimodal feature matrix is ​​obtained; Based on each trimodal synergy matrix and the target deep semantic feature matrix corresponding to each trimodal synergy matrix, a trimodal attention matrix corresponding to each trimodal synergy matrix is ​​obtained; Based on each trimodal attention matrix and the bimodal feature matrix corresponding to each trimodal attention matrix, a trimodal collaborative attention matrix corresponding to each trimodal attention matrix is ​​obtained; All the three-modal collaborative attention matrices are concatenated to obtain the primary fusion feature matrix.

3. The multimodal data fusion method for sentiment analysis according to claim 2, characterized in that: When the deep semantic feature matrix included in the feature group is a text deep semantic feature matrix and a visual deep semantic feature matrix, based on the preset multi-head attention formula and the deep semantic feature matrix included in the feature group, the bimodal feature matrix corresponding to the feature group is determined, including: Obtaining a preset trainable weight matrix, and determining a projection feature matrix based on the preset trainable weight matrix and a deep semantic feature matrix included in the feature group, wherein the preset trainable weight matrix includes a query trainable weight matrix, a key trainable weight matrix, and a value trainable weight matrix, The projection feature matrix includes a query key matrix corresponding to the text depth semantic feature matrix, a key key matrix and a value key matrix corresponding to the visual depth semantic feature matrix; Based on the preset multi-head attention formula, the query key matrix corresponding to the text deep semantic feature matrix, and the key key matrix and value key matrix corresponding to the visual deep semantic feature matrix, multiple attention scores corresponding to the feature group are determined. The preset multi-head attention formula is: , where Attention i is the i-th attention score corresponding to the feature group, i is a positive integer, and is the normalization factor, d k is the vector dimension of the key matrix corresponding to the visual depth semantic feature matrix; All attention scores corresponding to the feature group are concatenated to obtain a multi-head output result of the feature group, and a bimodal feature matrix corresponding to the feature group is determined based on the multi-head output result and a preset trainable linear transformation matrix.

4. The multimodal data fusion method for sentiment analysis according to claim 3, characterized in that: When the deep semantic feature matrix included in the feature group is a text deep semantic feature matrix and a visual deep semantic feature matrix, the step of determining the projection feature matrix based on the preset trainable weight matrix and the deep semantic feature matrix included in the feature group includes: Determine the query key matrix corresponding to the text deep semantic feature matrix according to the text deep semantic feature matrix, the query trainable weight matrix and the first calculation formula, wherein the first calculation formula is: , Q T is the query key matrix corresponding to the text deep semantic feature matrix, T is the text deep semantic feature matrix, W i Q To query the trainable weight matrix; According to the visual depth semantic feature matrix, the key trainable weight matrix and the second calculation formula, the key key matrix corresponding to the visual depth semantic feature matrix is ​​determined, and the second calculation formula is: , K V is the key matrix corresponding to the visual depth semantic feature matrix, V is the visual depth semantic feature matrix, W i K is the key trainable weight matrix; According to the visual depth semantic feature matrix, the value trainable weight matrix and the third calculation formula, the value key matrix corresponding to the visual depth semantic feature matrix is ​​determined, and the third calculation formula is: , V V is the value key matrix corresponding to the visual depth semantic feature matrix, V is the visual depth semantic feature matrix, W i V is the value of the trainable weight matrix; The query key matrix corresponding to the text depth semantic feature matrix, the key key matrix corresponding to the visual depth semantic feature matrix, and the value key matrix constitute a projection feature matrix.

5. The multimodal data fusion method for sentiment analysis according to claim 2, characterized in that: The at least three deep semantic feature matrices are subjected to feature-level fusion based on a preset multi-head attention formula to obtain a secondary fusion feature matrix, including: Concatenating at least three bimodal feature matrices to obtain a concatenated trimodal feature matrix; Performing dimensionality reduction processing on the spliced ​​trimodal feature matrix to obtain a reduced-dimensional spliced ​​trimodal feature matrix, and determining the self-attention distribution of the reduced-dimensional spliced ​​trimodal feature matrix based on a preset self-attention mechanism to obtain a self-attention spliced ​​trimodal feature matrix; The self-attention spliced ​​trimodal feature matrix and the at least three deep semantic feature matrices are spliced ​​to obtain a secondary fusion feature matrix.

6. The multimodal data fusion method for sentiment analysis according to claim 1, characterized in that: The step of performing decision-level fusion on the primary fusion feature matrix and the secondary fusion feature matrix to obtain a sentiment analysis result corresponding to the multimodal data within the target time period includes: Based on a preset classifier, the primary fusion feature matrix and the secondary fusion feature matrix are respectively classified to determine a target classification probability score; Based on the target classification probability score and the preset emotion category mapping relationship, determine the target emotion category corresponding to the target classification probability, wherein the preset emotion category mapping relationship is the corresponding relationship between the target classification probability and the target emotion category; Generate a sentiment analysis result based on the target sentiment category.

7. The multimodal data fusion method for sentiment analysis according to claim 6, characterized in that: The step of classifying the primary fusion feature matrix and the secondary fusion feature matrix based on a preset classifier to determine a target classification probability score includes: Obtaining classifier weight information and classifier bias information corresponding to the classifier, the classifier weight information including a first classifier weight and a second classifier weight, and the classifier bias information including a first classifier bias and a second classifier bias; The primary classification probability score corresponding to the primary fusion feature matrix is ​​determined based on the first classifier weight, the first classifier bias, the primary fusion feature matrix and a first classification formula, wherein the first classification formula is: , where l1 is the primary classification probability score, W1 is the first classifier weight, b1 is the first classifier bias, and P is the primary fusion feature matrix; The secondary classification probability score corresponding to the secondary fusion feature matrix is ​​determined based on the second classifier weight, the second classifier bias, the secondary fusion feature matrix and a second classification formula, wherein the second classification formula is: , where l2 is the secondary classification probability score, W2 is the second classifier weight, b2 is the second classifier bias, and S is the secondary fusion feature matrix; The target classification probability score is determined from the primary classification probability score and the secondary classification probability score based on a preset rule.

8. A fusion system, characterized in that: The fusion system includes: at least one processor; Memory; At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute a multimodal data fusion method for sentiment analysis according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: include: A computer program is stored which can be loaded by a processor and execute a multimodal data fusion method for sentiment analysis as described in any one of claims 1 to 7.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the steps of a multimodal data fusion method for sentiment analysis according to any one of claims 1 to 7.

Images