Feature fusion method, system and equipment for multi-modal data and medium

Abstract

The invention provides a feature fusion method, system and device for multi-modal data and a medium, and the method comprises the steps: obtaining text data and video data, carrying out the feature extraction of the text data and the video data, and obtaining a text feature and an image feature; dividing the text features and the image features into a plurality of text segments and a plurality of image segments respectively; constructing a memory unit, and generating historical information of each text segment or image segment by using the memory unit; inputting all the text segments and the image segments and the corresponding historical information into a feature extraction model for feature extraction to obtain depth text features and depth image features; and fusing the depth text features and the depth image features to obtain fused features. According to the method, the problems that in the prior art, dynamic features with long-distance dependence and obvious spatial-temporal features are difficult to extract, and key information is lost due to simple feature splicing or weighted average during multi-modal feature fusion are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data fusion, and in particular, to a method, system, device and medium for feature fusion of multimodal data. Background Art

[0002] Currently, there are significant bottlenecks in the core algorithms and model designs of current human feature extraction technologies. For example, traditional convolutional neural networks are limited in capturing long-range dependencies and spatio-temporal features, and it is difficult to effectively model fine-grained features such as human postures and micro-expressions. Especially in the case of occlusion or low-light conditions, the performance drops sharply. Although the Transformer architecture has improved global feature extraction through the self-attention mechanism, its computational complexity grows quadratically with the input size. For example, when real-time processing of high-resolution video streams, it faces severe latency problems. In addition, the multimodal fusion technology has not yet solved the semantic alignment problem of heterogeneous data (such as vision, speech, text). Simple feature splicing or weighted averaging often loses key correlation information. When the data distribution of the target domain is quite different from that of the training set, it is difficult to make the subsequent learning algorithms remain robust. Summary of the Invention

[0003] In view of the deficiencies in the prior art, the present invention provides a method, system, device and medium for feature fusion of multimodal data, which solves the problems in the prior art of difficult extraction of dynamic features with long-range dependencies and obvious spatio-temporal features, and that simple feature splicing or weighted averaging often loses key correlation information during multimodal feature fusion.

[0004] According to an embodiment of the present invention, a method for feature fusion of multimodal data includes:

[0005] Obtain text data and video data, perform feature extraction on the text data and the video data to obtain text features and image features respectively;

[0006] Unify the dimensions of the text features and the image features, and then divide the text features and the image features into multiple text segments and multiple image segments respectively;

[0007] Construct a memory unit, and use the memory unit to generate historical information of each text segment or image segment according to the text features and the image features;

[0008] Input all the text segments, image segments and the corresponding historical information into a feature extraction model for feature extraction to obtain deep text features and deep image features;

[0009] Preferably, the method for feature extraction of text data includes:

[0010] Perform One-Hot encoding conversion on the text data to obtain a text sparse matrix;

[0011] The text sparse matrix is processed through a dimensionality reduction matrix and then input into the fully connected layer to obtain text information;

[0012] The position encoder is used to perform position encoding on the text data to obtain temporal information;

[0013] The text information and the position information are added together to obtain text features.

[0014] Preferably, the method for extracting features from image data includes

[0015] Using the FFmpeg framework to extract key frames of the video data, then cropping all the key frames and adjusting all the key frames to the same size;

[0016] Using the RsnNet-18 model to extract features from all the key frames to obtain image features.

[0017] Preferably, the method for using the memory unit to generate historical information for each text segment or image segment includes:

[0018] S1: Perform dimensionality reduction and summation on the first dimension of the text segment or image segment to obtain a dimensionality reduction segment;

[0019] S2: Horizontally expand on the first dimension of the dimensionality reduction segment, and then use a multi-layer perceptron for processing to obtain initial information, and use the initial information as historical information;

[0020] S3: Starting from the first text segment or image segment, calculate the correlation degree between the current text segment or image segment and all historical information, and then calculate the candidate update information between the current text segment or image segment and the historical information according to the correlation degree;

[0021] S4: Sequentially select multiple candidate update information for merging to obtain the historical information corresponding to the current text segment or image segment;

[0022] S5: Traverse all text segments or image segments in sequence, repeating steps S3 - S4 to obtain the historical information corresponding to all text segments or image segments.

[0023] Preferably, the memory unit includes a reset gate and an update gate. The reset gate is used to calculate the correlation degree between the current text segment or image segment and all historical information in S3. If the correlation degree is less than a preset threshold, the historical information is directly eliminated;

[0024] The update gate is used to calculate the number of candidate update information for merging in S4.

[0025] Preferably, the intermediate fusion method is adopted to fuse the deep text feature and the deep image feature. The method for obtaining the fusion feature includes:

[0026] Use two independent multi-layer perceptrons to respectively increase the dimensions of the deep text feature and the deep image feature, and then splice the dimension-increased deep text feature and the deep image feature to obtain a spliced feature;

[0027] Use another multi-layer perceptron to reduce the dimension of the spliced feature to obtain a fusion feature.

[0028] Preferably, there are multiple memory units and feature extraction models, and they are alternately connected. The output of the memory unit is used as the input of the feature extraction model, the output of the feature extraction model is used as the input of the next memory unit, and the output of the last feature extraction model is fused to obtain a fusion feature.

[0029] On the other hand, according to the embodiments of the present invention, a feature fusion system for multi-modal data is also provided. This system uses the above-mentioned feature fusion method for multi-modal data, and includes:

[0030] An acquisition module, which is used to acquire text data and video data;

[0031] A preprocessing module, which is used to extract the features of the text data and the video data to obtain text features and image features respectively, and then unify the dimensions of the text features and the image features;

[0032] An encoding module, which is used to divide the dimension-unified text features and image features into multiple text segments and multiple image segments respectively, calculate the historical information of the text segments and the image segments, and then perform feature extraction according to the text segments, the image segments and the corresponding historical information to obtain deep text features and deep image features;

[0033] A fusion module, which is used to fuse the deep text features and the deep image features to obtain a fusion feature.

[0034] On the other hand, according to the embodiments of the present invention, a computer is also provided, including at least one processor and a memory. The memory stores a computer program, and the computer program is configured to be executed by the processor to implement the above-mentioned feature fusion method for multi-modal data.

[0035] On the other hand, according to the embodiments of the present invention, a storage medium is also provided. The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. The computer program can be executed by one or more processors to implement the above-mentioned feature fusion method for multi-modal data.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention segments text features and image features with long-range dependencies and obvious spatio-temporal characteristics to form text segments and image segments with short data lengths and small data sizes, reducing the computational complexity during feature extraction. Meanwhile, a memory unit is used to record the historical information between each segment, establish the context connection between each segment, strengthen the spatio-temporal characteristics between each segment, and avoid losing key information during feature extraction. Then, during feature fusion, an intermediate fusion method of first dimension elevation, then splicing, and finally dimension reduction is used for feature fusion to avoid direct splicing and loss of feature correlation information. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flowchart of feature fusion according to an embodiment of the present invention.

[0039] Figure 2 It is a flowchart of text data extraction according to an embodiment of the present invention.

[0040] Figure 3 It is an architecture diagram of a Transformer model with a memory unit according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] The technical solutions in the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] As Figure 1 shown, an embodiment of the present invention proposes a feature fusion method for multimodal data, including:

[0043] Obtain text data and video data, perform feature extraction on the text data and video data, and obtain text features and image features respectively;

[0044] As Figure 2 shown, the extraction process of text data is as follows:

[0045] (1) After obtaining the text vector, perform One-Hot encoding conversion on it according to the vocabulary size.

[0046] Specifically, a sparse matrix with a dimension of w_len×vocab_size will be obtained. In this sparse matrix, each word only corresponds to one position in the vocabulary, and only the value at that corresponding position is marked as 1, and the rest of the positions are 0.

[0047] For example, assume that the vocabulary size vocab_size = 1000 and the maximum text length w_len = 20. For a specific word, in the corresponding w_len rows of the sparse matrix, only one position in the vocab_size columns is 1, and the other 999 positions are 0.

[0048] After completing the One-Hot encoding, the obtained sparse matrix is processed by a dimensionality reduction matrix.

[0049] The role of the dimensionality reduction matrix is to compress the sparse matrix, reducing its dimension from w_len×vocab_size to w_len×d w Then, through a fully connected layer, the feature representation of the text is output.

[0050] This process can be calculated by the following formula:

[0051] F emb = MLP(W map × one_hot(w))

[0052] where W map is the weight for mapping the sparse matrix to the dense matrix, one_hot represents the one-hot encoding mapping, and F emb is the extracted word embedding encoding.

[0053] (3) Since a structure similar to a convolutional neural network is not used when extracting the text information representation and the temporal information of the text sequence cannot be extracted, therefore, the PostionalEncoding structure of Transformer is introduced during the text feature representation process.

[0054] This structure uses sine and cosine functions with different frequencies to perform positional encoding on the text to obtain the temporal information of the text. At the same time, because iterative model structures (such as RNN, LSTM, etc.) are not added to this model, parallel computing can be performed, accelerating the training speed of the model.

[0055] The calculation formula for the positional encoding is as follows:

[0056]

[0057] where pos represents the position of the word in the sequence, i represents the dimension, and d w is the dimension of the model. According to the formula, the finally obtained positional encoding representation is F pos .

[0058] After that, the positional encoding F pos is added to the previously obtained word embedding encoding F emb to obtain the final text feature representation A i = {A1, A2,..., A w_len}}.

[0059] The process of extracting image features is as follows:

[0060] (1) Considering that there are too many video frames in the video, if features are extracted from all video frames and input into the model, it will lead to an excessive computational load on the model and even cause a memory overflow problem. Therefore, the FFmpeg framework is used to extract key frames from the video. The specific operation is to take one frame every fixed duration, which can not only reduce the number of video frames but also not affect the overall content of the video. For example, for a video with a duration of 60 seconds and a frame rate of 30 fps, if the fixed duration is set to 1 second, then 60 frames will be extracted as key frames, greatly reducing the amount of data to be processed subsequently compared to the original 1800 frames.

[0061] (2) Since the picture lengths and widths of different videos are inconsistent, in order to uniformly input them into the pre-trained model to extract features, the obtained key frames need to be cropped to a size of 224×224. This operation ensures that all input pictures have the same size, facilitating subsequent model processing. For example, for a key frame picture with a length and width of 800×600, it is adjusted to a size of 224×224 through the cropping operation.

[0062] (3) Select the ResNet-18 pre-trained model as the picture feature extraction model. The ResNet model mainly solves the problem that as the number of network layers deepens, the model's ability to fit the labels decreases, resulting in a decline in the final training effect of the network.

[0063] The ResNet-18 pre-trained model used in the present invention is trained on the ImageNet dataset and was originally used for picture classification tasks. Its last fully connected layer acts as a "classifier". Since the present invention only uses this model to extract picture features rather than perform classification tasks, when extracting features, only the feature representation of the layer before the fully connected layer of the ResNet-18 pre-trained model needs to be obtained. The overall calculation formula for this step is as follows:

[0064] F i =ResNet[:-1](cut224×224(compressor(i)))

[0065] where compressor represents the function for extracting key frames, cut represents the function for cropping size, and F i ={f1,f2,...,f p} is the extracted image feature.

[0066] Unify the dimensions of the text feature and the image feature, and then divide the text feature and the image feature into multiple text segments and multiple image segments respectively;

[0067] The input image feature is F i ={f1,f2,...,f p}∈ER p×d v , where the dimension d v depends on the feature extraction method used and the nature of the image itself. The text feature is A i = {A1, A2,..., A w_len} ∈ R w_len ×d w , the number of words w_len and the dimension d w are related to the text content and the word embedding model adopted.

[0068] To unify the dimensions of video features and text features, two fully connected layers are constructed. The first fully connected layer is used to process image features, with a weight matrix w1 and a bias b1; the second fully connected layer is used to process text features, with a weight matrix w2 and a bias b2. These weight matrices and biases are optimized through the backpropagation algorithm during model training to adapt to different multimodal data distributions.

[0069] For the video feature F i , according to the formula F i ′ = w1F i + b1 for calculation, and its dimension is converted from p×d v to p×d h . During actual calculation, each row of the image feature matrix F i (representing the feature vector of a video frame) is multiplied by the weight matrix w1, and then the bias b1 is added to obtain the converted video feature F i ′.

[0070] For the text feature A i , according to the formula A i ′ = w2A i + b2 for conversion, and the dimension is changed from w_len×d w to w_len×d h . Similarly, each row of the text feature matrix A i (corresponding to the feature vector of a word) is multiplied by the weight matrix w2 and the bias b2 is added. After such dimension unification operations, the video and text features are consistent in dimension, which provides convenience for subsequent joint analysis and processing.

[0071] In an actual multi-modal data processing scenario, when the Transformer model processes long sequence data, due to its limited encoding ability for overly long contexts, it is prone to losing long-distance information. For example, when processing a continuous sequence of video frames or a long text, as the sequence length increases, it becomes difficult to effectively associate the information in the subsequent part with the previous content. To alleviate this problem, a segment mechanism is introduced to divide the text features and image features of the long sequence into multiple shorter text segments and multiple image segments respectively, reducing the amount of information input into the Transformer model each time.

[0072] Construct a memory unit. According to the text features and image features, use the memory unit to generate the historical information of each text segment or image segment;

[0073] Regarding the text features and image features, although the problem of the limited encoding ability of the Transformer model for overly long sequence data is solved, each segment becomes relatively independent, and the context connection between segments is cut off. Therefore, based on the existing structure, the present invention adds a memory unit structure to realize saving historical information through an external storage unit and updating historical information using the memory unit, thereby establishing the context connection between segments.

[0074] The input data of the memory unit is F = {z1, z2,..., z λ}, where F represents the text features or image features after segmentation. During the processing of the memory unit, only one segment is selected and input into the memory unit for calculation each time.

[0075] When the first segment is input for calculation, since there is no historical information in the external storage unit, an initialization operation is required to generate the initial historical information to be input into the model. Taking (x represents the number of video frames p or the input text length w_len) as an example, first perform a dimensionality reduction and summation operation on its first dimension to obtain a result with a dimension of Then, horizontally expand the result on the first dimension, and the expansion multiple is the number of cells O of the memory unit (O is a hyperparameter for model training. The larger O is, the larger the matrix dimension output by the memory unit and the larger the proportion of historical information input into the encoding block). The expanded dimension becomes Finally, input the expanded result into a multi-layer perceptron (MLP) for calculation to obtain the initialized occurrence information. When the first segment is input subsequently, the historical information M -1 obtained from the external storage unit is the initial information. In addition, create a set M -1 which contains the historical information corresponding to all text segments or image segments. At the initial moment, the set M -1 has only one initial information.

[0076] For any text segment or image segment z i , retrieve all historical information from the set M -1 , concatenate it with the current input segment z i to obtain the result o, and then use z i as the query input, o as the key input and value input, and perform an encoding operation through the multi-head attention mechanism to obtain the encoding result, which is used as the input of the next feature encoding block and participates in subsequent feature processing.

[0077] The memory unit is mainly used to generate historical information of text features or image features. It includes a reset gate and an update gate. The reset gate is used to calculate the correlation between the current text segment or image segment z i and all historical information. If the correlation is less than a preset threshold, the historical information is directly eliminated. First, use the attention mechanism to calculate the similarity Q -1 between the current text segment or image segment and all historical information in the set M i , and then calculate the correlation between the current text segment or image segment and all historical information:

[0078]

[0079] where both w1 and w2 are coefficients, Q i is the similarity between the text segment or image segment z i and the i-th historical information, b1 is the bias. The closer the correlation is to 0, the smaller the relationship between the text segment or image segment and the historical information. Therefore, if the correlation is less than 0.1, it is directly determined that the text segment or image segment has no relationship with the historical information and is directly discarded.

[0080] The update gate is used to generate historical information. Calculate the candidate update information between the current text segment or image segment z i and all historical information according to the similarity:

[0081]

[0082] where both w3 and w4 are coefficients, b2 is the bias, is the i-th historical information in the set M -1 , and Q i is the similarity between the text segment or image segment z i and the i-th historical information.

[0083] After that, according to the following formula, select some candidate update information to generate the historical information corresponding to the current text segment or image segment z i :

[0084]

[0085] Among them, w5 and w6 are coefficients, and b3 is the bias.

[0086] like Figure 3 As shown in the figure, all text fragments, image fragments and corresponding historical information are input into the feature extraction model for feature extraction. The feature extraction model used directly uses the feature extraction model in the Transformer model. In addition, a multi-layer coding block structure is used to process text features and image features. The output of each coding block will be used as the input of the next coding block to achieve deep encoding of features. Each coding block contains a memory unit and a feature extraction module. The output of the memory unit is used as the input of the feature extraction model, and the output of the feature extraction model is used as the input of the next memory unit. The output of the feature extraction model of the last coding block is fused to obtain deep text features and deep image features.

[0087] The intermediate fusion method is used to fuse the deep text features and deep image features to obtain the fused features.

[0088] In multimodal data processing, there are three types of model-independent multimodal fusion methods: early fusion, intermediate fusion, and late fusion. Early fusion is to fuse multiple independent modal information into an independent vector first, and then input it into the model for training; late fusion is to fuse the modal information through the respective models, and then through decision trees and other methods; and intermediate fusion is to input it into a unified model for training after the modal information is calculated by the respective models. Research has found that compared with early fusion and late fusion, intermediate fusion has the advantages of less information loss, lower fusion difficulty, and higher fusion level. Therefore, the present invention adopts the intermediate fusion method to finally fuse the deep image features and deep text features.

[0089] (1) Feature dimensionality upgrade:

[0090] Two independent multi-layer perceptrons (MLPs) are used to perform dimensionality increase operations on the deep image feature V0 and the deep text feature T0, respectively, converting V0 and T0 into high-dimensional feature expressions V0′ and T0′. The multi-layer perceptron is a feedforward artificial neural network model that can learn complex nonlinear relationships between input and output. The purpose of the dimensionality increase operation is to enable the model to learn more feature information and explore deeper semantic relationships in the data.

[0091] (2) Feature stitching:

[0092] The upsampled video feature V0' and text feature T0' are concatenated into a feature vector. The concatenation operation can integrate the feature information of the two modalities, providing more comprehensive information for subsequent fusion calculations.

[0093] (3) Feature downsampling:

[0094] Another multi-layer perceptron is used to perform a downsampling operation on the concatenated feature vector to obtain the fused feature S of multi-modal fusion. The downsampling operation can reduce the dimension of the features, remove redundant information, and at the same time retain the key feature information, making the fused features more suitable for downstream tasks.

[0095] On the other hand, an embodiment of the present invention also provides a feature fusion system for multi-modal data. The system uses the above-mentioned feature fusion method for multi-modal data, including:

[0096] An acquisition module, which is used to acquire text data and video data;

[0097] A preprocessing module, which is used to extract the features of the text data and video data, respectively obtain the text features and image features, and then unify the dimensions of the text features and image features;

[0098] An encoding module, which is used to divide the dimension-unified text features and image features into multiple text segments and multiple image segments respectively, calculate the historical information of the text segments and image segments, and then perform feature extraction according to the text segments, image segments and the corresponding historical information to obtain deep text features and deep image features;

[0099] A fusion module, which is used to fuse the deep text features and deep image features to obtain the fused features.

[0100] On the other hand, an embodiment of the present invention also provides a computer, including at least one processor and a memory. The memory stores a computer program, and the computer program is configured to be executed by the processor to implement the above-mentioned feature fusion method for multi-modal data.

[0101] On the other hand, an embodiment of the present invention also provides a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and the computer program can be executed by one or more processors to implement the above-mentioned feature fusion method for multi-modal data.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A feature fusion method for multimodal data, characterized in that: Including: Obtain text data and video data, perform feature extraction on the text data and video data respectively to obtain text features and image features; Unify the dimensions of the text features and image features, and then divide the text features and image features into multiple text segments and multiple image segments respectively; Construct a memory unit, and use the memory unit to generate historical information for each text segment or image segment according to the text features and image features; Input all text segments, image segments and their corresponding historical information into a feature extraction model for feature extraction to obtain deep text features and deep image features; Adopt an intermediate fusion method to fuse the deep text features and deep image features to obtain fusion features.

2. A feature fusion method for multi-modal data according to claim 1, characterized in that: The method for performing feature extraction on text data includes: Perform One-Hot encoding conversion on the text data to obtain a text sparse matrix; Process the text sparse matrix through a dimensionality reduction matrix and then input it into a fully connected layer to obtain text information; Use a position encoder to perform position encoding on the text data to obtain temporal information; Add the text information and the position information to obtain text features.

3. A feature fusion method for multi-modal data according to claim 1, characterized in that: The method for performing feature extraction on image data includes Use the FFmpeg framework to extract key frames of the video data, then crop all the key frames and adjust all the key frames to the same size; Use the RsnNet-18 model to perform feature extraction on all the key frames to obtain image features.

4. A feature fusion method for multi-modal data according to claim 1, characterized in that: The method for using a memory unit to generate historical information for each text segment or image segment includes: S1: Perform dimensionality reduction summation on the first dimension of the text segment or image segment to obtain a dimensionality reduction segment; S2: Horizontally expand on the first dimension of the dimensionality reduction segment, and then use a multi-layer perceptron for processing to obtain initial information, and use the initial information as historical information; S3: Starting from the first text segment or image segment, calculate the correlation degree between the current text segment or image segment and all historical information, and then calculate the candidate update information between the current text segment or image segment and the historical information according to the correlation degree; S4: Sequentially select multiple candidate update information for merging to obtain the historical information corresponding to the current text segment or image segment; S5: Traverse all text segments or image segments in sequence, and repeat steps S3-S4 to obtain the historical information corresponding to all text segments or image segments.

5. A feature fusion method for multi-modal data according to claim 4, characterized in that: The memory unit includes a reset gate and an update gate. The reset gate is used to calculate the correlation degree between the current text segment or image segment and all historical information in S3. If the correlation degree is less than a preset threshold, directly eliminate the historical information; The update gate is used to calculate the number of candidate update information for merging in S4.

6. The feature fusion method of multimodal data according to claim 1, wherein: The intermediate fusion method is adopted to fuse the deep text feature and the deep image feature. The method for obtaining the fused feature includes: Two independent multi-layer perceptrons are used to respectively increase the dimensions of the deep text feature and the deep image feature, and then the deep text feature and the deep image feature after dimension increase are concatenated to obtain a concatenated feature; Another multi-layer perceptron is used to reduce the dimension of the concatenated feature to obtain a fused feature.

7. The feature fusion method of multimodal data according to claim 6, wherein: There are multiple memory units and feature extraction models which are alternately connected. The output of the memory unit is used as the input of the feature extraction model, the output of the feature extraction model is used as the input of the next memory unit, and the output of the last feature extraction model is fused to obtain a fused feature.

8. A feature fusion system for multimodal data, characterized in that: The system uses the feature fusion method of multimodal data according to any one of claims 1-7, including: An acquisition module, which is used to acquire text data and video data; A preprocessing module, which is used to extract the features of the text data and the video data to obtain text features and image features respectively, and then unify the dimensions of the text features and the image features; An encoding module, which is used to divide the text features and the image features with unified dimensions into multiple text segments and multiple image segments respectively, calculate the historical information of the text segments and the image segments, and then extract features according to the text segments, the image segments and the corresponding historical information to obtain deep text features and deep image features; A fusion module, which is used to fuse the deep text features and the deep image features to obtain a fused feature.

9. A computer, characterized in that: It includes at least one processor and a memory. The memory stores a computer program, and the computer program is configured to be executed by the processor to implement the feature fusion method of multimodal data according to any one of claims 1-7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. The computer program can be executed by one or more processors to implement the feature fusion method of multimodal data according to any one of claims 1-7.

Images