Multi-modal feature level fusion method based on real-time sharing Transform

Abstract

The invention relates to a multi-modal feature level fusion method based on a real-time shared Transform, and relates to the technical field of multi-modal data processing. The method comprises the following steps: respectively transmitting image information, text information and voice information to a mapping module to obtain embedded vectors of three coding slices; the embedded vectors of the three kinds of coding slices are respectively transmitted to a common transform coding module, and three kinds of coding tokens are obtained; the three kinds of coding tokens are simultaneously transmitted to a real-time sharing transform coding module, and three kinds of sharing codes are obtained; the three kinds of shared codes are transmitted into an adaptive fusion module to obtain a joint feature representation containing all modal information. According to the method, multi-modal data are converted into embedded vectors, cross-modal information sharing is carried out after feature extraction and coding are carried out, and finally information of all modals is integrated in an adaptive fusion module, so that multi-modal information is fused more comprehensively and organically.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data processing, and in particular to a multimodal feature-level fusion method based on real-time shared Transformer. Background Art

[0002] With the rapid development of artificial intelligence, multimodal data processing technology has played an important role in the fusion of different modalities such as images, text, and speech. Traditional multimodal processing methods usually process the data of each modality independently, and then perform simple splicing or weighted fusion in the later stage. However, this method is prone to information imbalance problems, especially when there is a lack of real-time sharing and interaction between multimodal features, and the information advantages of each modality cannot be fully utilized.

[0003] Therefore, when processing multimodal data, existing technologies face the challenge of how to effectively utilize the complementary features of each modality in the fusion process to improve the overall processing effect. Summary of the invention

[0004] Based on this, it is necessary to provide a multimodal feature-level fusion method based on real-time shared Transformer to address the above problems.

[0005] The present invention provides a multimodal feature-level fusion method based on real-time shared Transformer, the method comprising:

[0006] The image information, text information, and voice information are respectively passed into the mapping module for slicing and vectorization processing to obtain the embedding vectors of the image coding slice, text coding slice, and voice coding slice;

[0007] The embedding vectors of the image coding slice, text coding slice and speech coding slice are respectively passed into the common transformer coding module to obtain the image coding token, text coding token and speech coding token;

[0008] The image encoding token, the text encoding token and the speech encoding token are simultaneously passed into the real-time shared transformer encoding module to obtain the shared image encoding, the shared text encoding and the shared audio encoding;

[0009] The shared image code, shared text code, and shared audio code are passed into the adaptive fusion module to obtain a joint feature representation containing all modal information.

[0010] In one embodiment, the steps of respectively transferring the image information, the text information and the voice information to the mapping module for slicing and vectorization processing include:

[0011] Segment the image information, text information, and voice information to obtain image coding slices, text coding slices, and voice coding slices;

[0012] The image coding slice, the text coding slice and the speech coding slice are flattened to obtain the embedding vectors of the image coding slice, the text coding slice and the speech coding slice.

[0013] In one embodiment, the steps of respectively passing the embedding vectors of the image coding slice, the text coding slice and the speech coding slice into the common transformer coding module to obtain the image coding token, the text coding token and the speech coding token include:

[0014] The embedding vectors of the image coding slice, the text coding slice, and the speech coding slice are processed by a multi-head self-attention mechanism to obtain a comprehensive attention output corresponding to multiple tokens;

[0015] The comprehensive attention outputs corresponding to multiple tokens are input into a feedforward neural network to obtain image encoding tokens, text encoding tokens, and speech encoding tokens.

[0016] In one embodiment, the step of processing the embedding vectors of the image coding slice, the text coding slice, and the speech coding slice with a multi-head self-attention mechanism to obtain respective comprehensive attention outputs includes:

[0017] Generate a query vector, a key vector, and a value vector for the embedding vectors of the image coding slice, the text coding slice, and the speech coding slice;

[0018] The generated query vector, key vector, and value vector are divided into multiple attention heads respectively, and multiple attention heads are calculated in parallel to obtain the attention weights of multiple attention heads;

[0019] Use the attention weights of multiple attention heads and perform weighted summation according to their corresponding value vectors to obtain the output of each attention head;

[0020] The output of each attention head is residually connected to the original input token and normalized through the layers to obtain the comprehensive attention output corresponding to multiple tokens.

[0021] In one embodiment, the steps of inputting the integrated attention outputs corresponding to the plurality of tokens into a feedforward neural network to obtain image encoding tokens, text encoding tokens and speech encoding tokens include:

[0022] The feedforward neural network performs linear transformation and nonlinear activation on the comprehensive attention outputs corresponding to multiple tokens to obtain the transformed feature representation;

[0023] The transformed feature representations are subjected to residual connections and layer normalization to finally obtain their respective encoding tokens.

[0024] In one embodiment, the steps of simultaneously passing the image encoding token, the text encoding token and the voice encoding token to the real-time shared transformer encoding module to carry out real-time shared encoding to obtain the shared image encoding, the shared text encoding and the shared audio encoding include:

[0025] Sharing the encoding features of the image modality to the text modality and the audio modality, sharing the encoding features of the text modality to the image modality and the audio modality, and sharing the encoding features of the audio modality to the image modality and the text modality through the first real-time sharing layer, to obtain image encoding tokens, text encoding tokens, and audio encoding tokens;

[0026] The image coding token, the text coding token and the audio coding token are encoded through the Transformer Block of the second real-time sharing layer to obtain a shared image code, a shared text code and a shared audio code.

[0027] In one of the embodiments, the first real-time shared layer includes three Transformer Blocks and one channel superposition point.

[0028] In one of the embodiments, the second real-time shared layer includes three Transformer Blocks, and each Transformer Bloc includes an encoder in a vision transformer.

[0029] In one embodiment, the step of passing the shared image code, the shared text code, and the shared audio code into an adaptive fusion module to obtain a joint feature representation containing all modal information includes:

[0030] A clustering algorithm is used to cluster the input shared image code, shared text code and shared audio code, and M types of category feature clusters are output;

[0031] Adopting adaptive fusion method to fuse the feature vectors of different modes within each cluster into a unified feature vector, generating M fused feature vectors;

[0032] The M fused feature vectors are superimposed to generate a single multimodal fused feature vector.

[0033] The present invention also provides a multi-modal feature-level fusion system based on real-time shared Transformer, the system comprising:

[0034] The mapping module, including slicing operations and vectorization processing, is used to process image information, text information, and speech information to obtain the embedding vectors of their respective modalities;

[0035] A common transformer encoding module, including a Transformer Encoder encoder, for obtaining common encoding tokens of three modalities according to the embedding vectors of the three modalities;

[0036] A real-time shared transformer encoding module, including a real-time sharing layer and a Transformer Block, is used to obtain encoding tokens shared in real time among the three modalities from common encoding tokens of the three modalities;

[0037] The adaptive fusion module includes a feature clustering layer, a first autonomous fusion operation and a second fusion operation, and is used to obtain multimodal fusion information according to the coded tokens shared in real time among the three modalities.

[0038] The above-mentioned multimodal feature-level fusion method based on real-time shared Transformer converts multimodal data (image, text, speech) into embedded vectors, then extracts and encodes features through the Transformer encoding module, then shares cross-modal information in the real-time shared Transformer encoding module, and finally integrates the information of all modalities in the adaptive fusion module to obtain a comprehensive feature representation, thereby achieving a more comprehensive and organic fusion of multimodal information, and learning more fine-grained feature information. The fused features generated subsequently can also integrate the features of multimodal data for a variety of downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A flowchart of a multimodal feature-level fusion method based on real-time shared Transformer proposed in the present invention;

[0041] Figure 2 This is a schematic diagram of the structure of the multimodal feature-level fusion system based on real-time shared Transformer proposed by the present invention;

[0042] Figure 3 for Figure 2 Schematic diagram of the mapping module;

[0043] Figure 4 for Figure 2 A schematic diagram of a first encoding module in FIG.

[0044] Figure 5 for Figure 2 Schematic diagram of real-time sharing module in ;

[0045] Figure 6 for Figure 2 Schematic diagram of the adaptive module. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] With the rapid development of the field of artificial intelligence, multimodal data processing technology has played an important role in the fusion of different modalities such as images, text, and speech. Traditional multimodal processing methods usually process the data of each modality independently, and then perform simple splicing or weighted fusion in the later stage. However, this method is prone to information imbalance problems, especially when there is a lack of real-time sharing and interaction between multimodal features, resulting in the inability of features between different modalities to complement each other during the processing process, reducing the performance of the overall model and failing to give full play to the information advantages of each modality.

[0048] At the same time, existing methods usually have difficulty in adaptively adjusting the weights of feature information between modalities, which results in a specific modality dominating the fusion process and suppressing the contribution of other modalities. These problems make it difficult for existing multimodal fusion methods to meet the requirements of high accuracy and high robustness in complex application scenarios.

[0049] In view of the above problems, the following Figure 1-Figure 6 The present invention describes the multimodal feature-level fusion based on real-time shared Transformer.

[0050] like Figure 1 As shown, in one embodiment, a multimodal feature-level fusion method based on real-time shared Transformer includes:

[0051] Step S100, the image information, text information, and voice information are respectively input into the mapping module, and slice operations and vectorization processing are performed to obtain the embedding vectors of the image coding slices, text coding slices, and voice coding slices.

[0052] Step S200, the embedding vectors of the image coding slice, the text coding slice and the speech coding slice are respectively passed into the common transformer coding module to obtain the image coding token, the text coding token and the speech coding token.

[0053] Step S300, the image coding token, the text coding token and the speech coding token are simultaneously passed into the real-time shared transformer coding module to obtain the shared image coding, the shared text coding and the shared audio coding.

[0054] Step S400: passing the shared image code, the shared text code and the shared audio code into an adaptive fusion module to obtain a joint feature representation containing all modal information.

[0055] Specifically, the image information, text information and voice information are respectively passed into the mapping module, and the information of each modality is sliced ​​to divide it into smaller units or fragments. Then, vectorization processing is applied to each slice to convert it into a vector of fixed dimension, thereby obtaining the embedding vectors of the image coding slice, text coding slice and voice coding slice. The above embedding vectors are then respectively passed into the ordinary Transformer encoding module. The Transformer encoding module extracts and transforms features through the self-attention mechanism and the feedforward neural network layer, and finally outputs the image coding token, text coding token and voice coding token.

[0056] The encoding tokens are simultaneously passed into the real-time shared Transformer encoding module, and the encoding tokens of different modalities are jointly processed to capture the association and interaction information between the modalities, generating shared image encoding, shared text encoding, and shared audio encoding. Finally, the shared image encoding, shared text encoding, and shared audio encoding are passed into the adaptive fusion module to integrate the encoding information from different modalities to form a joint feature representation containing all modal information.

[0057] See also Figure 3 In one embodiment, the steps of respectively transferring the image information, the text information and the voice information to the mapping module and performing the slicing operation and the vectorization processing include:

[0058] Step S110, segmenting the image information, text information, and voice information to obtain image coding slices, text coding slices, and voice coding slices.

[0059] Step S120, flatten the image coding slice, the text coding slice and the speech coding slice to obtain the embedding vectors of the image coding slice, the text coding slice and the speech coding slice.

[0060] Specifically, the steps for processing the image information include:

[0061] The input two-dimensional image is divided into multiple small blocks (patches). For example, if the image size is 224x224 pixels, the size of each patch can be 16x16 pixels. A 224x224 image can be divided into 196 16x16 small blocks.

[0062] Then a convolutional neural network (CNN) is applied to each image block to convert it into a feature vector of fixed dimension. The obtained feature vector is the encoded slice of the image.

[0063] The steps for processing text information include:

[0064] The input text sequence is divided into multiple subsequences or words, for example, a text is segmented into sentences or words. Then, the word embedding technology is applied to each subsequence or word to convert it into a vector of fixed dimension. The obtained vector is the encoded slice of the text.

[0065] The steps for processing voice information include:

[0066] The input speech signal is divided into multiple short-time frames. The acoustic feature extraction method is applied to each short-time frame to extract features, and the obtained feature vector is the coded slice of the speech.

[0067] In the process of obtaining the embedding vectors of the image coding slice, the text coding slice and the speech coding slice, the image coding slice, the text coding slice and the speech coding slice are flattened and converted into one-dimensional vectors. For example, the coding slice of each image block is a three-dimensional tensor of 4x4x64, so after flattening, it becomes a one-dimensional vector with a length of 1024.

[0068] See also Figure 4 In one embodiment, the steps of respectively passing the embedding vectors of the image coding slice, the text coding slice and the speech coding slice into the common transformer coding module to obtain the image coding token, the text coding token and the speech coding token include:

[0069] Step S210, the embedding vectors of the image coding slice, the text coding slice and the speech coding slice are processed by a multi-head self-attention mechanism to obtain a comprehensive attention output corresponding to multiple tokens.

[0070] Step S220, input the comprehensive attention outputs corresponding to the multiple tokens into a feedforward neural network to obtain image coding tokens, text coding tokens and speech coding tokens.

[0071] In this embodiment, the steps of performing multi-head self-attention processing on the embedding vectors of the image coding slice, the text coding slice, and the speech coding slice to obtain respective comprehensive attention outputs include:

[0072] Step S212, generating a query vector, a key vector, and a value vector for the embedding vectors of the image coding slice, the text coding slice, and the speech coding slice.

[0073] In step S214, the generated query vector, key vector, and value vector are divided into multiple attention heads respectively, and the multiple attention heads are calculated in parallel to obtain the attention weights of the multiple attention heads.

[0074] For example, if we set h heads, then each vector will be split into h parts, and the size of each part is 1 / h of the original size.

[0075] Specifically, for each head, the dot product between the query vector and all key vectors is calculated independently to obtain an unnormalized attention score matrix, the inverse square root of the key vector dimension is used as the scaling factor to scale the dot product result, and the softmax function is applied to the scaled attention score to obtain the normalized attention weight matrix.

[0076] Step S216, using the attention weights of multiple attention heads, perform weighted summation according to their corresponding value vectors to obtain the output of each attention head.

[0077] Specifically, the normalized attention weight matrix obtained for each head is used to perform weighted summation on the corresponding value vector to obtain the output representation of the head.

[0078] Step S218, the output of each attention head and the output of the multi-head attention are residually connected to the original input token, and layer normalization is performed to obtain a comprehensive attention output corresponding to multiple tokens.

[0079] Specifically, the multimodal fusion features generated by the fusion of the technical solution of the present invention, combined with multiple attention heads, can be freely deployed according to different task requirements to solve the problem of missing partial information of a single modality. It can provide complete and comprehensive multimodal feature information for downstream tasks such as classification tasks, detection tasks, segmentation tasks, etc. in various multimodal scenarios, thereby improving the training effect of downstream tasks.

[0080] In this embodiment, the steps of inputting the integrated attention outputs corresponding to the multiple tokens into a feedforward neural network to obtain image encoding tokens, text encoding tokens and speech encoding tokens include:

[0081] In step S222, the feedforward neural network performs linear transformation and nonlinear activation on the comprehensive attention outputs corresponding to multiple tokens to obtain a converted feature representation.

[0082] Step S224, perform residual connection and layer normalization processing on the converted feature representation to finally obtain the respective encoding tokens.

[0083] Specifically, first, the comprehensive attention output input corresponding to multiple tokens is transformed through the first linear transformation to obtain a new representation. The new representation is transformed through a nonlinear activation function (such as ReLU) to introduce nonlinear characteristics and obtain a richer and more complex new feature representation. The new feature representation is transformed again through the second linear transformation to map the representation of the high-dimensional space back to the original dimension to obtain the converted feature representation. After the converted feature representation is processed by residual connection and layer normalization, each token will get a new representation, namely the encoded token.

[0084] In the above mapping process, the second linear transformation weights and combines different features according to the weight matrix, so that some features may be strengthened, while others may be weakened or ignored. This selection mechanism enables the network to adaptively learn which features are more important for the current task.

[0085] See also Figure 5 In one embodiment, the steps of simultaneously passing the image encoding token, the text encoding token and the voice encoding token to the real-time shared transformer encoding module to carry out real-time shared encoding to obtain the shared image encoding, the shared text encoding and the shared audio encoding include:

[0086] Step S310, through the first real-time sharing layer, the coding features of the image modality are shared to the text modality and the audio modality, the coding features of the text modality are shared to the image modality and the audio modality, and the coding features of the audio modality are shared to the image modality and the text modality, to obtain image coding tokens, text coding tokens and audio coding tokens.

[0087] It should be noted that there are a total of i first real-time sharing layers, which means that feature sharing between modalities needs to be performed i times. The input of the i-th first real-time sharing layer is determined by the output of the (i-1)-th first real-time sharing layer. The output of the (i-1)-th first real-time sharing layer contains the encoding tokens of the three modalities at this stage.

[0088] Step S320 , encoding the image coding token, the text coding token and the audio coding token through the TransformerBlock of the second real-time shared layer to obtain a shared image code, a shared text code and a shared audio code.

[0089] Specifically, by sharing the encoded tokens of the three input modalities in real time at the feature level, the three modalities can achieve feature complementarity, avoid the problem of feature imbalance, and better integrate the feature information of other modalities.

[0090] In this embodiment, the first real-time sharing layer includes three Transformer Blocks and one channel superposition point.

[0091] In this embodiment, the second real-time shared layer includes three Transformer Blocks, and each TransformerBloc includes an encoder in a vision transformer.

[0092] Specifically, the Vision Transformer Encoder in the Transformer Block has significant differences from the traditional Transformer Encoder in terms of input representation, application of self-attention mechanism, position encoding, etc. These differences enable the Vision Transformer to process image data more effectively.

[0093] The present invention realizes the parallel feature extraction effect of multiple modalities through a modified real-time shared encoding module proposed based on Vision Transformer, solves the problem that Vision Transformer can only accept a single modality for feature extraction learning, makes the extracted information richer, and the information of each modality can be complementary.

[0094] See also Figure 6 In one embodiment, the steps of passing the shared image code, the shared text code and the shared audio code into an adaptive fusion module to obtain a joint feature representation containing all modal information include:

[0095] Step S410: clustering the input shared image code, shared text code and shared audio code using a clustering algorithm, and outputting M types of category feature clusters.

[0096] Optionally, the clustering algorithm is K-Means or hierarchical clustering or others.

[0097] Step S420: Adopt an adaptive fusion method to fuse the feature vectors of different modes within each cluster into a unified feature vector, and generate M fused feature vectors.

[0098] Optionally, the adaptive fusion method is weighted averaging or attention mechanism and others.

[0099] Step S430: superimpose the M fused feature vectors to generate a single multimodal fused feature vector.

[0100] Optionally, the superposition processing is splicing or summing or weighted superposition or others.

[0101] Specifically, the feature clustering layer performs feature clustering on the input three modal feature information and divides it into M categories, each category is composed of feature information from different modalities. The specific operation is as follows:

[0102] Receive the three modal encoding tokens from the common transformer encoding module, use an appropriate clustering algorithm to cluster the input features, and determine M categories. Each category contains feature information from different modalities, ensuring the diversity and complementarity of cross-modal features. Finally, output M feature clusters, each containing feature vectors from three modalities.

[0103] The first autonomous fusion operation fuses the feature information of different modes within each cluster to form a feature vector expression. A total of M autonomous fusion operations are required. The specific steps include:

[0104] Each feature cluster (a total of M) is processed independently category by category. Within each cluster, feature vectors from different modalities are identified and fused into a unified feature vector using weighted averaging, attention mechanism or other adaptive fusion methods.

[0105] The second fusion operation performs vector superposition on the M feature information after M autonomous fusion operations to obtain the feature information after multimodal fusion. The specific steps are as follows:

[0106] The M fusion feature vectors are superimposed according to certain strategies (such as splicing, summation, weighted superposition, etc.) to generate a comprehensive multimodal fusion feature vector, which represents the integrated multimodal information.

[0107] The multimodal information after the above integration will be provided to the subsequent learning or decision-making modules.

[0108] like Figure 2 As shown, in one embodiment, the present invention also provides a multimodal feature-level fusion system based on real-time shared Transformer, the system comprising:

[0109] The mapping module 10 includes slicing operations and vectorization processing, and is used to process image information, text information, and speech information to obtain embedding vectors of respective modalities.

[0110] The common transformer encoding module 20 includes a Transformer Encoder encoder, which is used to obtain common encoding tokens of three modalities according to the embedding vectors of the three modalities.

[0111] The real-time shared transformer encoding module 30 includes a real-time sharing layer and a Transformer Block, which is used to obtain encoding tokens shared in real time among the three modalities from common encoding tokens of the three modalities.

[0112] The adaptive fusion module 40 includes a feature clustering layer, a first autonomous fusion operation and a second fusion operation, and is used to obtain multimodal fusion information according to the coded tokens shared in real time among the three modalities.

[0113] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0114] The above-described embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A multimodal feature-level fusion method based on real-time shared Transformer, characterized in that: The method comprises: The image information, text information and speech information are respectively passed into the mapping module for slicing and vectorization processing to obtain the embedding vectors of the image coding slice, text coding slice and speech coding slice; The embedding vectors of the image coding slice, text coding slice and speech coding slice are respectively passed into the common transformer coding module to obtain the image coding token, text coding token and speech coding token; The image encoding token, the text encoding token and the speech encoding token are simultaneously passed into the real-time shared transformer encoding module to obtain the shared image encoding, the shared text encoding and the shared audio encoding; The shared image code, shared text code, and shared audio code are passed into the adaptive fusion module to obtain a joint feature representation containing all modal information.

2. The multimodal feature-level fusion method based on real-time shared Transformer according to claim 1 is characterized in that: The steps of respectively transferring the image information, text information and voice information to the mapping module for slicing operation and vectorization processing include: Segment the image information, text information, and voice information to obtain image coding slices, text coding slices, and voice coding slices; The image coding slice, the text coding slice and the speech coding slice are flattened to obtain the embedding vectors of the image coding slice, the text coding slice and the speech coding slice.

3. The multimodal feature-level fusion method based on real-time shared Transformer according to claim 1 is characterized in that: The steps of respectively passing the embedding vectors of the image coding slice, the text coding slice and the speech coding slice into the common transformer coding module to obtain the image coding token, the text coding token and the speech coding token include: The embedding vectors of the image coding slice, the text coding slice, and the speech coding slice are processed by a multi-head self-attention mechanism to obtain a comprehensive attention output corresponding to multiple tokens; The comprehensive attention outputs corresponding to multiple tokens are input into a feedforward neural network to obtain image encoding tokens, text encoding tokens, and speech encoding tokens.

4. The multimodal feature-level fusion method based on real-time shared Transformer according to claim 3 is characterized in that: The steps of processing the embedding vectors of the image coding slice, the text coding slice, and the speech coding slice through a multi-head self-attention mechanism to obtain their respective comprehensive attention outputs include: Generate a query vector, a key vector, and a value vector for the embedding vectors of the image coding slice, the text coding slice, and the speech coding slice; The generated query vector, key vector, and value vector are divided into multiple attention heads respectively, and multiple attention heads are calculated in parallel to obtain the attention weights of multiple attention heads; Use the attention weights of multiple attention heads and perform weighted summation according to their corresponding value vectors to obtain the output of each attention head; The output of each attention head is residually connected to the original input token and normalized through the layers to obtain the comprehensive attention output corresponding to multiple tokens.

5. The multimodal feature-level fusion method based on real-time shared Transformer according to claim 4 is characterized in that: The steps of inputting the comprehensive attention outputs corresponding to the multiple tokens into the feedforward neural network to obtain the image encoding token, the text encoding token and the speech encoding token include: The feedforward neural network performs linear transformation and nonlinear activation on the comprehensive attention outputs corresponding to multiple tokens to obtain the transformed feature representation; The transformed feature representations are subjected to residual connections and layer normalization to finally obtain their respective encoding tokens.

6. The multimodal feature-level fusion method based on real-time shared Transformer according to claim 1, characterized in that: The steps of simultaneously passing the image coding token, the text coding token and the speech coding token to the real-time shared transformer coding module to carry out real-time shared coding, and obtaining the shared image coding, the shared text coding and the shared audio coding include: Sharing the encoding features of the image modality to the text modality and the audio modality, sharing the encoding features of the text modality to the image modality and the audio modality, and sharing the encoding features of the audio modality to the image modality and the text modality through the first real-time sharing layer, to obtain image encoding tokens, text encoding tokens, and audio encoding tokens; The image coding token, the text coding token and the audio coding token are encoded through the Transformer Block of the second real-time sharing layer to obtain a shared image code, a shared text code and a shared audio code.

7. The multimodal feature-level fusion method based on real-time shared Transformer according to claim 6 is characterized in that: The first real-time sharing layer includes three Transformer Blocks and one channel superposition point.

8. The multimodal feature-level fusion method based on real-time shared Transformer according to claim 6, characterized in that: The second real-time shared layer includes three Transformer Blocks, and each Transformer Bloc includes an encoder in a vision transformer.

9. The multimodal feature-level fusion method based on real-time shared Transformer according to claim 1, characterized in that: The steps of passing the shared image code, shared text code and shared audio code into the adaptive fusion module to obtain a joint feature representation containing all modal information include: A clustering algorithm is used to cluster the input shared image code, shared text code and shared audio code, and M types of category feature clusters are output; Adopting adaptive fusion method to fuse the feature vectors of different modes within each cluster into a unified feature vector, generating M fused feature vectors; The M fused feature vectors are superimposed to generate a single multimodal fused feature vector.

10. A multimodal feature-level fusion system based on real-time shared Transformer, applied to a multimodal feature-level fusion method based on real-time shared Transformer according to any one of claims 1 to 9, characterized in that: The system comprises: The mapping module, including slicing operations and vectorization processing, is used to process image information, text information, and speech information to obtain the embedding vectors of their respective modalities; A common transformer encoding module, including a Transformer Encoder encoder, for obtaining common encoding tokens of three modalities according to the embedding vectors of the three modalities; A real-time shared transformer encoding module, including a real-time sharing layer and a Transformer Block, is used to obtain encoding tokens shared in real time among the three modalities from common encoding tokens of the three modalities; The adaptive fusion module includes a feature clustering layer, a first autonomous fusion operation and a second fusion operation, and is used to obtain multimodal fusion information according to the coded tokens shared in real time among the three modalities.

Images