Multimodal feature fusion method for intelligent systems based on dynamic sparse feature extraction

Abstract

The present application relates to a multimodal feature fusion method for an intelligent system based on dynamic sparse feature extraction, comprising: based on attention weights, weighted pooling of sparsely processed image features, speech features, and text features is performed through an attention mechanism, the dimensions of the pooled speech features and text features are mapped to a space with the same dimension as the pooled image features, and the spatial dimensions and number of channels are aligned; weighted image features are guided based on the aligned speech features and text features; the pooled image features are divided into multiple parts according to the spatial dimension, and the feature matching degree of each part is calculated based on the dimensions of the speech-guided and text-guided image features and each part; each part is selected for preliminary fusion with the speech-guided / text-guided image features based on the corresponding feature matching degree, and local fusion results of each part are obtained based on a gating mechanism, and the local fusion results of all parts are spliced ​​to obtain a multimodal fusion feature representation.

Description

Technical Field

[0001] The present application relates to the technical field of multimodal data fusion, and in particular to a multimodal feature fusion method for an intelligent system based on dynamic sparse feature extraction. Background Art

[0002] Multimodal data fusion technology, by integrating data sources from different modalities, can more comprehensively depict real-world scenarios and provide high-quality input support for various intelligent systems. However, traditional multimodal data processing methods often face challenges such as difficulty aligning heterogeneous data, ineffective fusion of intermodal features, and low computational efficiency. These issues hinder the further application and promotion of multimodal technology in complex human-computer interactions.

[0003] Specifically, the heterogeneity of multimodal data poses a significant challenge to cross-modal feature alignment. Data from different modalities exhibit significant differences in feature space, time scale, and information density. Traditional feature processing methods are unable to fully capture the intermodal relationships. Furthermore, existing feature fusion methods often rely on simple weighting or concatenation strategies, which can easily lead to information loss or excessive redundancy, thus compromising the system's decision-making capabilities and efficiency. Summary of the Invention

[0004] Based on this, it is necessary to provide a multimodal feature fusion method for intelligent systems based on dynamic sparse feature extraction, which includes:

[0005] S1: The intelligent system collects multimodal data including images, voice, and text, and performs preprocessing;

[0006] S2: Extract features from the preprocessed image, speech, and text, and perform sparse processing on the extracted image features, speech features, and text features;

[0007] S3: Based on the attention weights, the sparsely processed image features, speech features, and text features are weighted and pooled through the attention mechanism to obtain pooled image features, speech features, and text features;

[0008] S4: Map the dimensions of the pooled speech features and text features to the same space as the dimensions of the pooled image features, and align the spatial dimensions and number of channels;

[0009] S5: Based on the aligned speech features and text features, the weighted image features are guided respectively to obtain speech-guided and text-guided image features;

[0010] S6: Divide the pooled image features into multiple parts according to the spatial dimension, and calculate the feature matching degree of each part based on the voice-guided and text-guided image features and the dimensions of each part;

[0011] S7: Each part is preliminarily fused with the voice-guided / text-guided image features based on the corresponding feature matching degree, and the local fusion results of each part are obtained based on the gating mechanism. The local fusion results of all parts are spliced ​​together to obtain a multimodal fusion feature representation.

[0012] Beneficial effects: This method is different from traditional feature fusion methods. It does not rely on simple weighting or splicing strategies. Instead, it focuses on strengthening the intrinsic connections between multimodal data, deepening the deep interaction between images, voice, and text, and improving the efficiency and accuracy of multimodal fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0014] Figure 1 This is a flowchart of a multimodal feature fusion method for an intelligent system based on dynamic sparse feature extraction in an embodiment of the present application. DETAILED DESCRIPTION

[0015] To make the above-mentioned objects, features, and advantages of the present application more clearly understood, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0016] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0017] like Figure 1 As shown, this embodiment provides a multimodal feature fusion method for an intelligent system based on dynamic sparse feature extraction, the method comprising:

[0018] S1: The intelligent system collects multimodal data including images, voice, and text and performs preprocessing.

[0019] Specifically, image data is two-dimensional visual information acquired by a camera device. The acquisition frequency and resolution parameters are set to ensure that the image quality meets the requirements of subsequent processing. The spatial dimensions of the image data are the height and width of the image, and the number of channels of the image data is the number of color channels.

[0020] Voice data is the audio signal recorded by the microphone device, with the sampling rate and bit rate set to ensure the clarity and integrity of the voice signal; the audio signal is recorded synchronously with a timestamp.

[0021] Text data is obtained by converting speech data using speech recognition technology (such as deep neural network models). Converting speech data to text effectively separates content information, highlighting its importance and allowing subsequent processing of different modalities to focus on their key features. During this conversion process, the speech modality focuses on spectral features (such as duration and frequency), which help distinguish the urgency of tasks.

[0022] The pretreatment includes:

[0023] Synchronize image data, voice data, and text data through timestamps;

[0024] Perform denoising and normalization on image data, speech data, and text data.

[0025] After preprocessing, the data is stored in a multimodal database for retrieval and analysis; at the same time, quality control is implemented to eliminate low-quality data, providing a reliable data foundation for multimodal alignment and fusion of human-machine collaboration.

[0026] S2: Extract features from the preprocessed image, speech, and text respectively, and perform sparse processing on the extracted image features, speech features, and text features respectively.

[0027] Specifically, a convolutional neural network is used to extract features from the preprocessed image to obtain image features;

[0028] The acoustic model is used to extract features from the preprocessed speech to obtain speech features;

[0029] Use the pre-trained language model to extract features from the pre-processed text to obtain text features;

[0030] The image features, speech features and text features are respectively thinned out using a sparse function, and the image features, speech features and text features after thinning out are obtained.

[0031] This step helps retain the informative parts of the image while eliminating redundant features, providing more streamlined and efficient feature input for subsequent modeling; it can make text information more concentrated on key semantic units, thereby improving the efficiency of semantic transmission in multimodal fusion; it can effectively extract the most representative time series features from complex speech signals, avoiding the interference of redundant data, and at the same time strengthening the key information in the speech.

[0032] S3: Based on the attention weight, the sparsely processed image features, speech features, and text features are weighted pooled through the attention mechanism to obtain pooled image features, speech features, and text features.

[0033] Specifically, image data contains rich spatial information, and the importance of different regions in the task may vary greatly. Therefore, a spatial attention mechanism is used to dynamically select regions with prominent features in the image. The spatial attention mechanism assigns weights to each region of the image through a self-attention model, thereby guiding the network to focus on the most critical part for the current task.

[0034] Based on the spatial attention weight, the sparse image features are weighted by the spatial attention mechanism to obtain attention-weighted image features; the attention-weighted image features are subjected to maximum pooling and average pooling respectively to obtain maximum pooling image features and average pooling image features; the maximum pooling image features are multiplied by the weighting coefficient, and the average pooling image features are multiplied by a balance number of the weighting coefficient about 1, and the two multiplication results are added to obtain the pooled image features.

[0035] Speech data not only contains semantic information, but also time series features such as intonation, rhythm, and emotional fluctuations. In order to effectively capture the time periods in speech that are critical to the task, the temporal attention mechanism is used to calculate the weights of different time segments and dynamically select key information areas in the speech.

[0036] Based on the temporal attention weight, the sparsely processed speech features are weighted by the temporal attention mechanism to obtain attention-weighted speech features; the attention-weighted speech features are subjected to maximum pooling and average pooling respectively to obtain maximum pooling speech features and average pooling speech features; the maximum pooling speech features are multiplied by the weighting coefficient, and the average pooling speech features are multiplied by the balance number of the weighting coefficient about 1, and the two multiplication results are added to obtain the pooled speech features.

[0037] Text data usually contains a lot of redundant information, especially in long texts. Therefore, the text attention mechanism is used to focus on the most meaningful parts of the text for the task. The text attention mechanism dynamically assigns weights to each part by analyzing the importance of each word or sentence in the text.

[0038] Based on the text attention weight, the sparse text features are weighted through the text attention mechanism to obtain attention-weighted text features; the attention-weighted text features are subjected to maximum pooling and average pooling respectively to obtain maximum pooling text features and average pooling text features; the maximum pooling text features are multiplied by the weighting coefficient, and the average pooling text features are multiplied by the balance number of the weighting coefficient about 1, and the two multiplication results are added to obtain the pooled text features.

[0039] Through the weighted pooling method, the contribution ratio of maximum pooling and average pooling is dynamically adjusted to ensure that key information areas receive higher weights.

[0040] S4: Map the dimensions of the pooled speech features and text features to the same space as the dimensions of the pooled image features, and align the spatial dimensions and number of channels.

[0041] Specifically, through the fully connected layer, the dimensions of the pooled speech features and text features are mapped to the same space as the dimensions of the pooled image features;

[0042] The mapped speech features and text features are aligned in terms of spatial dimensions and number of channels through two-dimensional convolution operations.

[0043] The specific mathematical formula for the two-dimensional convolution operation is as follows:

[0044] Taking text features as an example, the shape of the text features after input mapping is (H in , W in , C in ), where: H in : The height of the input feature map; W in : The width of the input feature map; C in : The number of channels of the input feature map;

[0045] The number of convolution kernels is K, which is the number of output channels;

[0046] The size of the convolution kernel is (F h ,F w ), usually (3,3);

[0047] The stride is S;

[0048] The padding mode is 'same', that is, the height and width of the output feature map are the same as the input;

[0049] The mathematical expression of the convolution operation is:

[0050] For each position (i, j) and the kth output channel in the output aligned text feature, its value can be expressed as:

[0051] ;

[0052] in, Represents the value of each position (i, j) and the k-th output channel in the aligned text features; represents the weight of the k-th convolution kernel at position (m, n) on channel c; T represents the mapped text features; Represents the bias term of the kth convolution kernel; Indicates rounding up.

[0053] In this embodiment, the step further includes:

[0054] Generative adversarial networks are introduced to optimize alignment effects. Generative adversarial networks include:

[0055] Generator, used to generate pooled image features aligned with aligned speech features and text features, and calculate the generation loss;

[0056] The discriminator is used to evaluate the alignment between the generated pooled image features and the aligned speech features and text features, and calculate the discriminant loss;

[0057] The generator and the discriminator are trained adversarially until the generation loss and the discriminator loss converge and are balanced.

[0058] During adversarial learning, the generator optimizes the generated image features to align them with the information of text and speech, while the discriminator evaluates the difference between the generated image features and the real image features, thereby driving continuous improvement in generation.

[0059] S5: Based on the aligned speech features and text features, weighted image features are guided respectively to obtain speech-guided and text-guided image features.

[0060] Specifically, they include:

[0061] Multiply the aligned speech features with the speech attention weights and the pooled image features, and pass the first product through the softmax activation function to obtain the speech-condition guided attention weights.

[0062] Multiply the aligned text features with the text attention weights and the pooled image features, and pass the second product through the softmax activation function to obtain the text condition guided attention weights;

[0063] The above attention weights reflect the correlation between different modalities and are used to guide the focus of image features.

[0064] Multiply the speech-conditioned attention weights by the pooled image features element-wise to obtain the speech-guided image features;

[0065] The text-conditioned guided attention weights are element-wise multiplied with the pooled image features to obtain the text-guided image features.

[0066] The generated attention weights are applied to image features through an element-wise multiplication operation, highlighting image regions related to speech and text; in this way, image features can accurately align the semantics of speech and text, thereby improving the performance of multimodal tasks.

[0067] S6: Divide the pooled image features into multiple parts according to the spatial dimension, and calculate the feature matching degree of each part based on the voice-guided and text-guided image features and the dimensions of each part.

[0068] Specifically, they include:

[0069] Divide the pooled image features into multiple parts according to the spatial dimension;

[0070] For any part, the cosine similarity is calculated based on the image features of the voice guidance and the text guidance and the dimensions of the corresponding part, and the first feature matching degree corresponding to the voice guidance and the second feature matching degree corresponding to the text guidance are obtained;

[0071] Traverse all parts and obtain the first feature matching degree and the second feature matching degree corresponding to all parts.

[0072] S7: Each part is preliminarily fused with the voice-guided / text-guided image features based on the corresponding feature matching degree, and the local fusion results of each part are obtained based on the gating mechanism. The local fusion results of all parts are spliced ​​together to obtain a multimodal fusion feature representation.

[0073] By comparing the matching degree of speech-guided and text-guided features, the image data is dynamically segmented into parts related to the speech and text content respectively, and local features are fused according to the matching degree. Finally, the overall features are spliced ​​and fused to obtain a multimodal fusion feature representation.

[0074] Specifically, the aforementioned parts select and fuse the image features guided by voice or text based on the corresponding feature matching degree, and the initial fusion includes:

[0075] For any part,

[0076] Compare the size relationship between the first feature matching and the second feature matching,

[0077] When the first feature matching degree is greater than the second feature matching degree, the voice-guided image feature is selected and fused with the dimension of the corresponding part to obtain a first preliminary fusion result;

[0078] When the first feature matching degree is less than the second feature matching degree, the text-guided image feature is selected and fused with the dimension of the corresponding part to obtain a second preliminary fusion result;

[0079] When the first feature matching degree is equal to the second feature matching degree, the first preliminary fusion result is multiplied by the fusion coefficient, the second preliminary fusion result is multiplied by the balance number of the fusion coefficient with respect to 1, and the two multiplication results are added to obtain a third preliminary fusion result. In this embodiment, the fusion coefficient is 0.5.

[0080] Traverse all parts and obtain the first preliminary fusion result / second preliminary fusion result / third preliminary fusion result of all parts.

[0081] In this embodiment, the fusion operation uses a fusion function (such as a fusion function), a fusion model (such as a ResNet model) or a fusion method (such as a residual connection) commonly used in the art to fuse image features and dimensions.

[0082] Furthermore, the multimodal fusion feature representation includes:

[0083] For any part,

[0084] The dimensions of the corresponding parts and the image features guided by the speech are subjected to a gating mechanism to obtain a first gating coefficient;

[0085] The dimensions of the corresponding parts and the image features guided by the text are subjected to a gating mechanism to obtain a second gating coefficient;

[0086] The dimensions of the corresponding parts are simultaneously combined with the image features guided by speech and text through a gating mechanism to obtain the third gating coefficient;

[0087] When the corresponding part is fused with the speech-guided image feature, the first gating coefficient is element-wise multiplied by the first preliminary fusion result, the dimension of the corresponding part is element-wise multiplied by the balance number of the first gating coefficient about 1, and the two multiplication results are added to obtain a first local fusion result;

[0088] When the corresponding part is fused with the text-guided image feature, the second gating coefficient is element-wise multiplied by the second preliminary fusion result, the dimension of the corresponding part is element-wise multiplied by the balance number of the second gating coefficient about 1, and the two multiplication results are added to obtain a second local fusion result;

[0089] When the corresponding part is fused with the voice-guided and text-guided image features at the same time, the third gating coefficient is element-wise multiplied by the third preliminary fusion result, the dimension of the corresponding part is element-wise multiplied by the balance number of the third gating coefficient about 1, and the two multiplication results are added to obtain a third local fusion result;

[0090] Traverse all parts and obtain the first local fusion result / second local fusion result / third local fusion result of all parts;

[0091] The first local fusion results / second local fusion results / third local fusion results of all parts are spliced ​​together to obtain a multimodal fusion feature representation.

[0092] After obtaining the final multimodal fusion feature representation, deep learning models are used to further mine the information within it to arrive at the optimal decision. For example, deep neural networks (such as multilayer perceptrons, recurrent neural networks and their variants, and convolutional neural networks) are employed to process the multimodal fusion feature representation. These deep learning models can learn the complex mapping relationships between different feature combinations and decision outcomes from extensive training data. This allows them to accurately output the optimal decision based on the input multimodal features for various tasks.

[0093] In terms of human-computer interaction, the decision outputs of deep learning models provide strong support for the interactive process. The system can generate corresponding interactive content based on the model's predictions. For example, in a voice assistant scenario, the system generates natural and fluent voice responses that align with the user's intent based on the fusion of voice, text, and image information. It also displays relevant visual content (if applicable) in conjunction with image information to enhance the intuitiveness and effectiveness of the interaction.

[0094] Furthermore, deep learning models are capable of self-optimization based on user feedback. During human-computer interaction, the system collects user feedback on decision-making outcomes (such as corrections, supplementary explanations, and satisfaction ratings). This feedback is then used to fine-tune the model's parameters via a backpropagation algorithm, thereby continuously improving model performance and optimizing multimodal fusion strategies. This results in more accurate subsequent decision-making, further enhancing the quality and efficiency of human-computer interaction.

[0095] The multimodal feature fusion method for intelligent systems based on dynamic sparse feature extraction provided in this embodiment has the following beneficial effects:

[0096] 1. Efficiently extract image, speech, and text features through sparsification, removing redundant information and retaining key information. Image features preserve the two-dimensional spatial structure, text features focus on semantic keywords, and speech features capture the temporal characteristics of emotions or key segments, laying the foundation for subsequent modeling.

[0097] 2. A weighted pooling method is added to dynamically adjust the contribution ratio of maximum pooling and average pooling. This method can adaptively combine the features of different modalities, enhance the model's sensitivity to specific modal information, and improve the efficiency and accuracy of multimodal fusion.

[0098] 3. By using speech and text features as the guide for attention distribution of image features, and comparing the matching degree of speech and text features with image features, this method achieves dynamic adjustment and local fusion of image features. This further strengthens the deep interaction between image, speech, and text, avoiding over-reliance on global alignment, making it particularly suitable for real-time interaction and resource-constrained application scenarios.

[0099] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A multimodal feature fusion method for intelligent systems based on dynamic sparse feature extraction, characterized in that: include: S1: The intelligent system collects multimodal data including images, voice, and text, and performs preprocessing; S2: Extract features from the preprocessed image, speech, and text, and perform sparse processing on the extracted image features, speech features, and text features; S3: Based on the attention weights, the sparsely processed image features, speech features, and text features are weighted and pooled through the attention mechanism to obtain pooled image features, speech features, and text features; S4: Map the dimensions of the pooled speech features and text features to the same space as the dimensions of the pooled image features, and align the spatial dimensions and number of channels; S5: Based on the aligned speech features and text features, the weighted image features are guided respectively to obtain speech-guided and text-guided image features; S6: Divide the pooled image features into multiple parts according to the spatial dimension, and calculate the feature matching degree of each part based on the voice-guided and text-guided image features and the dimensions of each part; S7: Each part is preliminarily fused with the voice-guided / text-guided image features based on the corresponding feature matching degree, and the local fusion results of each part are obtained based on the gating mechanism. The local fusion results of all parts are spliced ​​together to obtain a multimodal fusion feature representation.

2. The multimodal feature fusion method for intelligent systems based on dynamic sparse feature extraction according to claim 1, characterized in that: The pretreatment includes: Synchronize image data, voice data, and text data through timestamps; Perform denoising and normalization on image data, speech data, and text data.

3. The multimodal feature fusion method for intelligent systems based on dynamic sparse feature extraction according to claim 1, characterized in that S2 include: Use convolutional neural network to extract features from preprocessed images to obtain image features; The acoustic model is used to extract features from the preprocessed speech to obtain speech features; Use the pre-trained language model to extract features from the pre-processed text to obtain text features; The image features, speech features and text features are respectively thinned out using a sparse function, and the image features, speech features and text features after thinning out are obtained.

4. The multimodal feature fusion method for intelligent systems based on dynamic sparse feature extraction according to claim 1, characterized in that S3 include: Based on the spatial attention weight, the sparse image features are weighted through the spatial attention mechanism to obtain the attention-weighted image features; The attention-weighted image features are subjected to maximum pooling and average pooling respectively to obtain maximum pooling image features and average pooling image features; the maximum pooling image features are multiplied by the weighting coefficient, and the average pooling image features are multiplied by a balance number of the weighting coefficient about 1, and the two multiplication results are added to obtain the pooled image features; Based on the temporal attention weight, the sparsely processed speech features are weighted by the temporal attention mechanism to obtain attention-weighted speech features; the attention-weighted speech features are subjected to maximum pooling and average pooling respectively to obtain maximum pooling speech features and average pooling speech features; Multiply the maximum pooled speech feature by the weighting coefficient, multiply the average pooled speech feature by a balance number of the weighting coefficient about 1, and add the two multiplication results to obtain the pooled speech feature; Based on the text attention weight, the sparse text features are weighted through the text attention mechanism to obtain attention-weighted text features; the attention-weighted text features are subjected to maximum pooling and average pooling respectively to obtain maximum pooling text features and average pooling text features; the maximum pooling text features are multiplied by the weighting coefficient, and the average pooling text features are multiplied by the balance number of the weighting coefficient about 1, and the two multiplication results are added to obtain the pooled text features.

5. The multimodal feature fusion method for intelligent systems based on dynamic sparse feature extraction according to claim 1, characterized in that S4 include: Through the fully connected layer, the dimensions of the pooled speech features and text features are mapped to the same space as the dimensions of the pooled image features; The mapped speech features and text features are aligned in terms of spatial dimensions and number of channels through two-dimensional convolution operations.

6. The multimodal feature fusion method for intelligent systems based on dynamic sparse feature extraction according to claim 5, characterized in that: S4 also includes; Generative adversarial networks are introduced to optimize alignment effects. Generative adversarial networks include: Generator, used to generate pooled image features aligned with aligned speech features and text features, and calculate the generation loss; The discriminator is used to evaluate the alignment between the generated pooled image features and the aligned speech features and text features, and calculate the discriminant loss; The generator and the discriminator are trained adversarially until the generation loss and the discriminator loss converge and are balanced.

7. The multimodal feature fusion method for intelligent systems based on dynamic sparse feature extraction according to claim 1, characterized in that S5 include: Multiply the aligned speech features with the speech attention weights and the pooled image features, and pass the first product through the softmax activation function to obtain the speech-condition guided attention weights. Multiply the aligned text features with the text attention weights and the pooled image features, and pass the second product through the softmax activation function to obtain the text condition guided attention weights; Multiply the speech-conditioned attention weights by the pooled image features element-wise to obtain the speech-guided image features; The text-conditioned guided attention weights are element-wise multiplied with the pooled image features to obtain the text-guided image features.

8. The multimodal feature fusion method for intelligent systems based on dynamic sparse feature extraction according to claim 7, characterized in that S6 include: Divide the pooled image features into multiple parts according to the spatial dimension; For any part, the cosine similarity is calculated based on the image features of the voice guidance and the text guidance and the dimensions of the corresponding part, and the first feature matching degree corresponding to the voice guidance and the second feature matching degree corresponding to the text guidance are obtained; Traverse all parts and obtain the first feature matching degree and the second feature matching degree corresponding to all parts.

9. The multimodal feature fusion method for intelligent systems based on dynamic sparse feature extraction according to claim 8, characterized in that: The aforementioned parts are initially integrated with the voice-guided / text-guided image features based on the corresponding feature matching degree, including: For any part, Compare the size relationship between the first feature matching and the second feature matching, When the first feature matching degree is greater than the second feature matching degree, the voice-guided image feature is selected and fused with the dimension of the corresponding part to obtain a first preliminary fusion result; When the first feature matching degree is less than the second feature matching degree, the text-guided image feature is selected and fused with the dimension of the corresponding part to obtain a second preliminary fusion result; When the first feature matching degree is equal to the second feature matching degree, the first preliminary fusion result is multiplied by the fusion coefficient, the second preliminary fusion result is multiplied by the balance number of the fusion coefficient about 1, and the two multiplication results are added to obtain a third preliminary fusion result; Traverse all parts and obtain the first preliminary fusion result / second preliminary fusion result / third preliminary fusion result of all parts.

10. The multimodal feature fusion method for intelligent systems based on dynamic sparse feature extraction according to claim 9, characterized in that: The multimodal fusion feature representation obtained includes: For any part, The dimensions of the corresponding parts and the image features guided by the speech are subjected to a gating mechanism to obtain a first gating coefficient; The dimensions of the corresponding parts and the image features guided by the text are subjected to a gating mechanism to obtain a second gating coefficient; The dimensions of the corresponding parts are simultaneously combined with the image features guided by speech and text through a gating mechanism to obtain the third gating coefficient; When the corresponding part is fused with the speech-guided image feature, the first gating coefficient is element-wise multiplied by the first preliminary fusion result, the dimension of the corresponding part is element-wise multiplied by the balance number of the first gating coefficient about 1, and the two multiplication results are added to obtain a first local fusion result; When the corresponding part is fused with the text-guided image feature, the second gating coefficient is element-wise multiplied by the second preliminary fusion result, the dimension of the corresponding part is element-wise multiplied by the balance number of the second gating coefficient about 1, and the two multiplication results are added to obtain a second local fusion result; When the corresponding part is fused with the voice-guided and text-guided image features at the same time, the third gating coefficient is element-wise multiplied by the third preliminary fusion result, the dimension of the corresponding part is element-wise multiplied by the balance number of the third gating coefficient about 1, and the two multiplication results are added to obtain a third local fusion result; Traverse all parts and obtain the first local fusion result / second local fusion result / third local fusion result of all parts; The first local fusion results / second local fusion results / third local fusion results of all parts are spliced ​​together to obtain a multimodal fusion feature representation.

Images