Multi-vector data stream sentiment analysis method based on geometric algebra

Abstract

The invention relates to a geometric algebra-based multi-vector data stream sentiment analysis method, which comprises the following steps of: independently extracting sentiment characteristics of audio, text and image modalities to obtain multi-vector data streams, and fusing the multi-vector data streams by using an asymmetric multi-head attention mechanism dominated by a double-layer text; visual and auditory information most relevant to text sentiment content is concerned, vector data flow processed by an attention mechanism and modal missing codes of one-hot codes are expressed in a geometric algebraic form, and therefore accurate sentiment analysis is obtained. Compared with the prior art, the method has the advantages of high accuracy, inter-modal information balance and the like.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a multi-vector data stream sentiment analysis method based on geometric algebra. Background Art

[0002] With the rapid development of information technology, social media and mobile communication devices have rapidly become important platforms for modern people to express their emotions. Through these platforms, users use multiple modalities (such as text, images, videos, and audio) to convey their emotions. These multimodal data types enrich the way of communication and also provide new perspectives and tools for analyzing and understanding users' emotional states. Multi-vector data stream sentiment analysis technology is a technology that converts multimodal data such as text, images, videos, and audio into vector data streams for analyzing emotions. This technology can be used to have a more comprehensive and in-depth understanding of emotional expressions on social media and communication devices. This technology plays a key role in many application scenarios. For example, in recommendation systems, it can grasp users' emotional tendencies and preferences, thereby providing more personalized content recommendations; in social psychology research, it can be used to analyze and predict group emotional dynamics, providing a scientific basis for social management and decision-making; in the field of human-computer interaction, machines using multi-vector data stream sentiment analysis technology can better understand human emotional states and needs, making interactions more natural and user-friendly. Therefore, it is of great significance to study accurate multi-vector data stream sentiment analysis technology.

[0003] In the field of multi-vector data stream sentiment analysis, existing research focuses on how to more effectively integrate information from different sources. In traditional fusion methods, such as linear weighting and feature vector concatenation, the interactive dynamics and nonlinear relationships between different vector data streams are often ignored. To address these issues, a cross-vector data stream attention mechanism is introduced to interactively and dynamically integrate the information of various vector data streams while still maintaining uniqueness. However, this method does not completely solve the problem of uneven weight distribution between vector data streams in sentiment analysis. In terms of emotional expression, psychologists have verified that textual modalities contain more stable and richer semantic information and higher emotional expression capabilities than visual and auditory modalities. In addition, another technical difficulty is to align vector data stream data. For example, instances of unaligned multi-vector data stream data have been identified in the IEMOCAP dataset. When using attention weights for cross-vector data stream correlation analysis, the "sad" state in the text vector data stream is highly correlated with the "low pitch" in the auditory vector data stream and the "frown" expression in the visual vector data stream, but these vector data stream expressions do not always appear at the same time. In practical applications, missing data in vector data streams is a common problem, such as the inability to obtain complete visual features due to camera blind spots; background noise may interfere with the clarity of audio information; or in some scenarios, text information cannot be directly obtained to protect privacy. Although existing methods describe solutions to these problems from the perspective of spatial mapping, the contribution of text is still ignored in the fusion process. How to make full use of higher-weighted text information for data fusion to achieve higher emotion recognition accuracy has become a technical problem that needs to be solved. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a multi-vector data stream sentiment analysis method based on geometric algebra. By independently extracting the sentiment features of audio, text and image modalities, a multi-vector data stream is obtained and converted into geometric algebra, and a double-layer asymmetric fusion mechanism is used to fuse the data, so as to obtain accurate sentiment analysis.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A multi-vector data stream sentiment analysis method based on geometric algebra, combining geometric algebra with semantic enhanced attention mechanism, includes the following steps:

[0007] S1, single modal feature extraction, extracting the emotional features of visual modality, text modality and audio modality to obtain multi-vector data stream;

[0008] S2. Fusion of multiple vector data streams through a double-layer text-dominated asymmetric multi-head attention mechanism, focusing on the visual and auditory information most relevant to the emotional content of the text, and expressing the vector data stream processed by the attention mechanism in a geometric algebraic form;

[0009] S3. Encode the fusion features based on one-hot encoding. If there is modality loss input data, the first bit of the label is set to 0, otherwise it is 1. The last three bits of the label are used to indicate the corresponding visual, auditory and textual modalities. If there is a missing bit, the corresponding bit is set to 1, otherwise it is 0. The modality loss encoding is represented by E tag ;

[0010] S4, performing geometric algebraic expression on the vector data stream and the modal missing code based on one-hot encoding;

[0011] S5. Transformer performs encoding output.

[0012] As a preferred technical solution, in step S1, when extracting the visual modality, OpenFace 2 is used to extract features from the face, head and eyes; when extracting the text modality, the ERT model is used to represent the word vector to capture the emotional expression of the text; when extracting the audio modality, the zero crossing rate, Mel-frequency cepstral coefficients and constant Q transform spectrogram are extracted through the Librosa toolkit.

[0013] The two-layer asymmetric multi-head attention mechanism in step S2 is a cross-modal attention mechanism that enables the model to focus on the visual and auditory information most relevant to the emotional content of the text. The attention mechanism can be dynamically adjusted according to changes in the text content, thereby more accurately capturing relevant emotional expressions in other modalities. The standard multi-head attention mechanism fusion structure is shown in the following formula. The attention mechanism in the Base-M model is defined as follows:

[0014] Base-M=Multi(Q,K,V)=Concat(head1,...,headh)W O ,

[0015] Among them, each head i The calculation formula is:

[0016]

[0017] The attention function is defined as:

[0018]

[0019] Among them, Q, K and V represent query, key and value matrices respectively, and matrix W i Q ,W iK and is the weight matrix corresponding to the i-th head in the attention mechanism. O ) is the weight matrix of the output concatenation. k ) represents the dimension of the key vector. The softmax function is applied to the matrix product (QK T ) and through Scale the dot product to prevent the values ​​from becoming too large during the softmax calculation.

[0020] The emotional content in images is usually reflected at the frame level; auditory emotional features require understanding of contextual relationships. Therefore, their text mappings should not be placed on the same layer. In order to discover the best two-layer multi-head attention mechanism, a variety of different two-layer attention mechanisms are designed to explore the best structure. The first layer is used to process the interaction between text and visual modalities. The fusion mechanism is described as:

[0021]

[0022]

[0023]

[0024] Among them, Q v is the query of visual modality features, Q t Query of text modality features, Q a is the query of auditory modality features, K t is the key of the text modality feature, K a is the key of the auditory modality feature, V t is the value of the text modality feature, V a is the value of the auditory modality feature.

[0025] The second layer is used to process the interaction between text and audio modalities. Text features are mapped to audio to understand the contextual relationship of the audio modality. The fusion mechanism is described as follows:

[0026]

[0027]

[0028]

[0029] Among them, Q v is the query of visual modality features, Q t Query of text modality features, Q a is the query of auditory modality features, K v is the key of the visual modality feature, K t is the key of the text modality feature, K ais the key of the auditory modality feature, V v is the value of the visual modality feature, V t is the value of the text modality feature, V a is the value of the auditory modality feature.

[0030] In the process of information fusion, the multivariate data features are transferred to high-dimensional geometric algebraic space through dimensionality-upgrading transformation, and relevant features are sought in more feature spaces. Through dimensionality-upgrading transformation, the low-order features of the original vector data stream data are expanded into multi-level high-order geometric features in the geometric algebraic space, thereby making full use of the correlation information and high-order structure between the variables of the original vector data stream data.

[0031] The original vector data stream {E v e 1 ,E a e 2 ,E t e 3 After the dimensionality increase transformation, the characteristic expression of its geometric algebraic space is:

[0032] E all `=E v e 1 +E a e 2 +E t e 3 +E v E a e 1 e 2 +E a E t e 2 e 3 +E t E v e 3 e 1 +E v E a E t e 1 e 2 e 3 ,

[0033] Among them, E v is the visual modality information, E a is the auditory modality information, E t For text modal information.

[0034] The vector data stream also needs to be added with the modal loss encoding, expressed as:

[0035] E all =E tag +E all `

[0036] =Etag +E v e 1 +E a e 2 +E t e 3 +E v E a e 1 e 2 +E a E t e 2 e 3 +E t E v e 3 e 1 +E v E a E t e 1 e 2 e 3 .

[0037] Among them, E tag is the modal loss encoding, E v is the visual modality information, E a is the auditory modality information, E t For text modal information.

[0038] The original three-dimensional vector data stream is represented as a directed line segment in the three-dimensional Euclidean space, and after the dimensionality increase transformation, it is represented as a multivector in its geometric algebraic space, that is, a set of directed geometric entities (directed line segments, directed face elements, directed volume elements). The sample can be represented not only by a directed line segment, but also by directed face elements, etc.

[0039] In order to effectively model long-term dependencies within and across modalities, a new sublayer is integrated into the Transformer. At the same time, a composite training objective is defined, including label recovery loss, forward difference loss, and reverse reconstruction loss, to guide the learning process in the absence of modality.

[0040] Label recovery loss (L tag ): In this invention, labels are used to mark missing modalities and enhance the network's attention to them. In order to more accurately guide the reconstruction of additional labels, a label recovery loss is designed. Due to the robustness of the mean absolute error (MAE) to outliers, the encoder modality loss is calculated using MAE. tag and decoder loss encoding D tag The losses among the last four:

[0041]

[0042] Among them, D tag=Sigmoid(D out [-4:]), D out Transformer decoder output.

[0043] Forward difference loss (L forward ): This loss is based on the pre-training output (E pre ) and Transformer encoder output (E out ). The pre-trained model is trained with complete modality data, and the features of the three modalities are directly concatenated for classification, so the differential loss is used to guide the learning process of modality loss. The Kullback-Leibler (KL) divergence loss is used to measure, and due to the asymmetry of KL divergence, it is replaced by Jensen-Shannon (JS) divergence loss.

[0044] The reverse reconstruction loss (L backward ): This loss is used to supervise the reconstruction of the joint common vector. Similar to the forward difference loss, the Transformer decoder output (D out ) and the updated fused public representation (E all ) between JS divergence loss.

[0045] Use the Softmax classifier for emotion recognition and obtain the emotion classification results. This method calculates the probability distribution of a set of categories, ensuring that the sum of the probabilities of all categories is equal to 1. The Softmax function exponentially modifies each category input and normalizes these values ​​by the sum of the exponentials of all category inputs.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] (1) Improving the accuracy of sentiment analysis: Through the asymmetric two-layer multi-head attention mechanism, especially emphasizing the core role of text information in sentiment analysis, the accuracy of sentiment recognition is significantly improved.

[0048] (2) Solving the problem of information imbalance between modalities: The asymmetric two-layer multi-head attention mechanism effectively solves the instability of visual and auditory modalities and the problem of uneven information weights between modalities, allowing multimodal data to contribute information more evenly during the fusion process.

[0049] (3) Innovative application of geometric algebra: Innovatively integrate geometric algebra into model construction and use the idea of ​​geometric algebra to realize multi-modal vector data stream fusion. This method not only enriches the theoretical basis of multi-modal fusion, but also provides new ideas for subsequent research. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1This is a structural diagram of the multi-vector data stream sentiment analysis method based on geometric algebra;

[0051] Figure 2 This is a diagram of the structure of the asymmetric multi-head attention mechanism dominated by double-layer text;

[0052] Figure 3 It is the geometric meaning diagram of 3-vector;

[0053] Figure 4 Fusion flow chart for multiple vector data streams using geometric algebraic expressions;

[0054] Figure 5 It is a geometric meaning diagram of the three-dimensional vector data stream after the dimensionality increase transformation;

[0055] Figure 6 Schematic diagram of the model training process. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0057] A multi-vector data stream sentiment analysis method based on geometric algebra, combining geometric algebra with semantic enhanced attention mechanism, includes the following steps:

[0058] S1, single modal feature extraction, extracting the emotional features of visual modality, text modality and audio modality to obtain multi-vector data stream;

[0059] S2. Fusion of multiple vector data streams through a double-layer text-dominated asymmetric multi-head attention mechanism, focusing on the visual and auditory information most relevant to the emotional content of the text, and expressing the vector data stream processed by the attention mechanism in a geometric algebraic form;

[0060] S3. Encode the fusion features based on one-hot encoding. If there is modality loss input data, the first bit of the label is set to 0, otherwise it is 1. The last three bits of the label are used to indicate the corresponding visual, auditory and textual modalities. If there is a missing bit, the corresponding bit is set to 1, otherwise it is 0. The modality loss encoding is represented by E tag ;

[0061] S4, performing geometric algebraic expression on the vector data stream and the modal missing code based on one-hot encoding;

[0062] S5. Transformer performs encoding output.

[0063] like Figure 1 The structure diagram of the method of the present invention is shown. For example, if the auditory modality is missing, the label is set to "0010"; if both the visual and auditory modalities are missing, the label is set to "0110".

[0064] In step S1, when extracting the visual modality, OpenFace 2 is used to extract features from the face, head, and eyes; when extracting the text modality, the ERT model is used to represent the word vector to capture the emotional expression of the text; when extracting the audio modality, the Librosa toolkit is used to extract the zero-crossing rate, Mel-frequency cepstral coefficients, and constant Q transform spectrogram.

[0065] The two-layer asymmetric multi-head attention mechanism in step S2 is a cross-modal attention mechanism that enables the model to focus on the visual and auditory information most relevant to the emotional content of the text. The attention mechanism can be dynamically adjusted according to changes in the text content, thereby more accurately capturing relevant emotional expressions in other modalities. The standard multi-head attention mechanism fusion structure is shown in the following formula. The attention mechanism in the Base-M model is defined as follows:

[0066] Base-M=Multi(Q,K,V)=Concat(head1,...,headh)W O ,

[0067] Among them, each head i The calculation formula is:

[0068]

[0069] The attention function is defined as:

[0070]

[0071] Among them, Q, K and V represent query, key and value matrices respectively, and matrix W i Q ,W i K and is the weight matrix corresponding to the i-th head in the attention mechanism. O ) is the weight matrix of the output concatenation. k ) represents the dimension of the key vector. The softmax function is applied to the matrix product (QK T ) and through Scale the dot product to prevent the values ​​from becoming too large during the softmax calculation.

[0072] The emotional content in images is usually reflected at the frame level; auditory emotional features require understanding of contextual relationships. Therefore, their text mappings should not be placed on the same layer. In order to discover the best two-layer multi-head attention mechanism, a variety of different two-layer attention mechanisms are designed to explore the best structure, such as Figure 2 The first layer is used to process the interaction between text and visual modalities, and the fusion mechanism is described as:

[0073]

[0074]

[0075]

[0076] Among them, Q v is the query of visual modality features, Q t Query of text modality features, Q a is the query of auditory modality features, K t is the key of the text modality feature, K a is the key of the auditory modality feature, V t is the value of the text modality feature, V a is the value of the auditory modality feature.

[0077] The second layer is used to process the interaction between text and audio modalities. Text features are mapped to audio to understand the contextual relationship of the audio modality. The fusion mechanism is described as follows:

[0078]

[0079]

[0080]

[0081] Among them, Q v is the query of visual modality features, Q t Query of text modality features, Q a is the query of auditory modality features, K v is the key of the visual modality feature, K t is the key of the text modality feature, K a is the key of the auditory modality feature, V v is the value of the visual modality feature, V t is the value of the text modality feature, V a is the value of the auditory modality feature.

[0082] In the process of information fusion, the multivariate data features are transferred to high-dimensional geometric algebraic space through dimensionality-upgrading transformation, and relevant features are sought in more feature spaces. Through dimensionality-upgrading transformation, the low-order features of the original vector data stream data are expanded into multi-level high-order geometric features in the geometric algebraic space, thereby making full use of the correlation information and high-order structure between the variables of the original vector data stream data.

[0083] For a three-dimensional Euclidean space, let {e 1 ,e 2 ,e 3} is the standard basis of the three-dimensional vector space, and the geometric product of vectors a and b is defined as:

[0084] ab=a·b+a∧b

[0085] Here, a·b is a scalar multiplication, and its result is a scalar.

[0086] e 1 ,e 2 , the geometric product e 1 e 2 is a 2-vector or a plane vector, and e 1 ,e 2 ,e 3 The geometric product of 1 e 2 e 3 It is called a 3-vector. Note that the 3-vector here is different from the vector in the vector space. It is a mathematical quantity in geometric algebra. Figure 3 Shown is the geometric meaning of 3-vector, e 1 e 2 Indicates e 1 ,e 2 Oriented surface element, and e 1 ,e 2 ,e 3 Multiplying the three vectors gives e 1 e 2 e 3 Oriented voxel.

[0087] The original vector data stream {E v e 1 ,E a e 2 ,E t e 3 After the dimensionality increase transformation, the characteristic expression of its geometric algebraic space is:

[0088] E all `=E v e 1 +E a e 2 +E t e3 +E v E a e 1 e 2 +E a E t e 2 e 3 +E t E v e 3 e 1 +E v E a E t e 1 e 2 e 3 ,

[0089] Among them, E v is the visual modality information, E a is the auditory modality information, E t For text modal information.

[0090] like Figure 4 As shown, the vector data stream also needs to be added with the modal loss encoding, expressed as:

[0091] E all =E tag +E all `

[0092] =E tag +E v e 1 +E a e 2 +E t e 3 +E v E a e 1 e 2 +E a E t e 2 e 3 +E t E v e 3 e 1 +E v E a E t e 1 e 2 e 3

[0093] Among them, E tag is the modal loss encoding, E v is the visual modality information, E a is the auditory modality information, E t For text modal information.

[0094] like Figure 5 As shown in the figure, the original three-dimensional vector data stream is represented as a directed line segment in the three-dimensional Euclidean space, and after the dimensionality increase transformation, it is represented as a multi-vector in its geometric algebraic space, that is, a set of directed geometric entities (directed line segments, directed face elements, directed volume elements). The sample is not only represented by a directed line segment, but also by directed face elements, etc., which is the shaded area in the figure.

[0095] In order to effectively model long-term dependencies within and across modalities, a new sublayer is integrated into the Transformer. At the same time, a composite training objective is defined, including label recovery loss, forward difference loss, and reverse reconstruction loss, to guide the learning process in the absence of modality.

[0096] Label recovery loss (L tag ): In this invention, labels are used to mark missing modalities and enhance the network's attention to them. In order to more accurately guide the reconstruction of additional labels, a label recovery loss is designed. Due to the robustness of the mean absolute error (MAE) to outliers, the encoder modality loss is calculated using MAE. tag and decoder loss encoding D tag The losses among the last four:

[0097]

[0098] Among them, D tag =Sigmoid(D out [-4:]), D out Transformer decoder output.

[0099] Forward difference loss (L forward ):like Figure 1 As shown, the loss is based on the pre-training output (E pre ) and Transformer encoder output (E out ). The pre-trained model is trained with complete modality data, and the features of the three modalities are directly concatenated for classification, so the differential loss is used to guide the learning process of modality loss. The Kullback-Leibler (KL) divergence loss is used to measure,

[0100]

[0101] Where p and q both represent probabilities. Due to the asymmetry of KL divergence, it is replaced by Jensen-Shannon (JS) divergence loss.

[0102] The reverse reconstruction loss (L backward): This loss is used to supervise the reconstruction of the joint common vector. Similar to the forward difference loss, the Transformer decoder output (D out ) and the updated fused public representation (E all ) between JS divergence loss:

[0103]

[0104] Use the Softmax classifier for emotion recognition and obtain the emotion classification results. This method calculates the probability distribution of a set of categories, ensuring that the sum of the probabilities of all categories is equal to 1. The Softmax function exponentially modifies each category input and normalizes these values ​​by the sum of the exponentials of all category inputs.

[0105] like Figure 6 The schematic diagram of the model training process of the present invention is shown. Before formal training, the weights of the neural network are pre-trained, which helps the model converge to better performance faster and determines the total number of times the model needs to be trained. The vector data stream data from three different modes (which may be text, image, and audio) is used as the input of the model, and the input data is first preliminarily processed and analyzed by the fully connected layer. After the fully connected layer, a double-layer asymmetric multi-head attention mechanism is used to emphasize the core role of text information in sentiment analysis and improve the accuracy of sentiment recognition. The vector data streams from three different modes are fused to ensure that the model can make full use of the information in the multimodal data. The fused vector data stream is combined with the modal information based on unique hot encoding, and the data form is further converted to express the possible missing codes geometrically. The processed data is encoded and output using the Transformer model. The classification probability and various loss functions are calculated according to the output of the model, and the weights of the model are updated by the back propagation algorithm to optimize the model performance. In the back propagation process, the stochastic gradient descent algorithm is used to update the weights of the network and gradually approach the optimal solution. Finally, determine whether the current number of training times has reached the set upper limit. If so, stop training; otherwise, continue the training process. After the training is completed, save the trained model and softmax classifier for subsequent prediction or further analysis.

[0106] To verify the performance of the present invention, this embodiment conducted experiments on two public datasets (CMU-MOSI and IEMOCAP), and analyzed and compared a single-layer text-dominated multi-head attention mechanism network, a single-layer multi-head self-attention mechanism network, and the double-layer asymmetric text-dominated multi-head attention mechanism network of the present invention.

[0107] These results highlight the efficiency of the dual-layer asymmetric text-dominated multi-head attention mechanism model, which shows the highest accuracy and precision MF1 scores on the two datasets, reaching 88.63% / 90.89% and 68.82% / 89.06% respectively. This further emphasizes the key role of text modality in multimodal sentiment analysis, as it contains rich emotional and semantic content. Considering the characteristics of visual and auditory modalities, the dual-layer asymmetric text-dominated multi-head attention structure designed in this invention can capture the dynamic interaction between modalities in a more detailed manner, and effectively solve the problem of uneven information weights between different modalities, as well as innovatively represent multi-vector data streams by geometric algebra and perform deep fusion in geometric algebraic form.

[0108] Taking advantage of the text modality, a semantically enhanced two-layer multi-head attention mechanism is used to improve the overall semantic understanding ability of the model. This mechanism effectively integrates information from different modalities and enhances the accuracy and depth of sentiment analysis. The experimental results are shown in Tables 1, 2, and 3.

[0109] Table 1 Comparison of experimental data between CMU-MOSI and IEMOCAP

[0110]

[0111] Table 2 Comparison of experimental data of the model in the case of modal missing in the CMU-MOSI dataset

[0112]

[0113] Table 3 Comparison of experimental data of the model in the case of missing modes in the IEMOCAP dataset

[0114]

[0115]

[0116] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multi-vector data stream sentiment analysis method based on geometric algebra, characterized in that: The method combines geometric algebra with semantically enhanced attention mechanism, including the following steps: S1, single modal feature extraction, extracting the emotional features of visual modality, text modality and audio modality to obtain multi-vector data stream; S2. Fusion of multiple vector data streams through a double-layer text-dominated asymmetric multi-head attention mechanism, focusing on the visual and auditory information most relevant to the emotional content of the text, and expressing the vector data stream processed by the attention mechanism in a geometric algebraic form; S3. Encode the fusion features based on one-hot encoding. If there is modality loss input data, the first bit of the label is set to 0, otherwise it is 1. The last three bits of the label are used to indicate the corresponding visual, auditory and textual modalities. If there is a missing bit, the corresponding bit is set to 1, otherwise it is 0. The modality loss encoding is represented by E tag ; S4, performing geometric algebraic expression on the vector data stream and the modal missing code based on one-hot encoding; S5. Transformer performs encoding output.

2. According to the method of multi-vector data stream sentiment analysis based on geometric algebra in claim 1, it is characterized in that: In the step S1, when extracting the visual modality, OpenFace 2 is used to extract features from the face, head and eyes; when extracting the text modality, the ERT model is used to represent the word vector to capture the emotional expression of the text; when extracting the audio modality, the zero crossing rate, Mel frequency cepstral coefficient and constant Q transform spectrogram are extracted through the Librosa toolkit.

3. The method for multi-vector data stream sentiment analysis based on geometric algebra according to claim 1 is characterized in that: The first layer of the two-layer asymmetric multi-head attention mechanism in step S2 is used to process the interaction between text and visual modalities, and the second layer is used to process the interaction between text and audio modalities. The text features are mapped to audio to understand the contextual relationship of the audio modality.

4. The method for multi-vector data stream sentiment analysis based on geometric algebra according to claim 3 is characterized in that: The fusion mechanism of the first layer of the two-layer asymmetric multi-head attention mechanism is described as: Among them, Q v is the query of visual modality features, Q t Query of text modality features, Q a is the query of auditory modality features, K t is the key of the text modality feature, K a is the key of the auditory modality feature, V t is the value of the text modality feature, V a is the value of the auditory modality feature.

5. The method for multi-vector data stream sentiment analysis based on geometric algebra according to claim 3 is characterized in that: The fusion mechanism of the second layer of the two-layer asymmetric multi-head attention mechanism is described as: Among them, Q v is the query of visual modality features, Q t Query of text modality features, Q a is the query of auditory modality features, K v is the key of the visual modality feature, K t is the key of the text modality feature, K a is the key of the auditory modality feature, V v is the value of the visual modality feature, V t is the value of the text modality feature, V a is the value of the auditory modality feature.

6. The method for multi-vector data stream sentiment analysis based on geometric algebra according to claim 1 is characterized in that: The geometric algebraic expression transfers the multivariate data features to a high-dimensional geometric algebraic space through a dimensionality-raising transformation, including directed line segments, oriented face elements and oriented volume elements, and represents the association information and high-order structure between the data variables of the original vector data stream.

7. The method for multi-vector data stream sentiment analysis based on geometric algebra according to claim 6 is characterized in that: The original vector data stream {E v e1,E a e2,E t After the dimension-raising transformation, the characteristic expression of its geometric algebraic space is: AND all `=And v e1+E a e2+E t e3+E v AND a e1e2+E a AND t e2e3+E t AND v e3e1+E v AND a AND t e1e2e3, Among them, E v is the visual modality information, E a is the auditory modality information, E t For text modal information.

8. The method for multi-vector data stream sentiment analysis based on geometric algebra according to claim 7 is characterized in that: The vector data stream also needs to be added with the modal loss code, expressed as: AND all =And tag +E all ` =And tag +E v e1+E a e2+E t e3+E v AND a e1e2+E a AND t e2e3+E t AND v e3e1+E v AND a AND t e1e2e3。 Among them, E tag is the modal loss encoding, E v is the visual modality information, E a is the auditory modality information, E t For text modal information.

9. The method for multi-vector data stream sentiment analysis based on geometric algebra according to claim 1 is characterized in that It also includes defining a composite training objective, including label recovery loss, forward difference loss, and backward reconstruction loss, to guide the learning process in the absence of modality.

10. The method for multi-vector data stream sentiment analysis based on geometric algebra according to claim 1, characterized in that: The fused data is subjected to sentiment classification using a Softmax classifier, where each category input is indexed and normalized by the sum of the indexes of all category inputs.

Images