Multi-modal data fusion method based on meta learning
By constructing cross-modal sharing underlying encoder and meta-learning algorithm optimization parameters, the problem of weak computing resource consumption and generalization capabilities in multimodal data fusion is solved, and multimodal data fusion that quickly adapts to new tasks and new modes is achieved.
Patent Information
- Application Number
- CN202510699992.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-28
- Publication Date
- 2025-09-02
AI Technical Summary
It is difficult for the prior art to effectively integrate multimodal data, especially when dealing with data differences, heterogeneous data representations and new mode adaptation of different modes, there are problems such as loss of information, excessive computing resource consumption and weak generalization capabilities.
Using a meta-learning-based method, a cross-modal shared underlying encoder is built, and the initial parameters are trained and optimized across modal annotation data sets, model updates are updated using mixed data sets and pseudo-labels, shared parameters are frozen, and modal embedding layer is trained using small annotation samples to achieve fast task adaptation.
Reliance on a large amount of labeled data is reduced, computing resource consumption and training time is reduced, and the generalization ability and accuracy of the model under new tasks and new modes is improved.
Smart Images
Figure CN120579136A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of multimodal data fusion methods based on meta-learning, and more specifically to a multimodal data fusion method and device based on meta-learning. Background Art
[0002] With the continuous development and advancement of data acquisition technology, the application of multimodal data (images, text, speech, sensor data, etc.) is becoming increasingly common in various fields, such as medical imaging, autonomous driving, intelligent security, and smart city construction. In these scenarios, due to the differences in characteristics, dimensions, and representation methods of multimodal data, how to effectively utilize and fuse this multimodal data to obtain more accurate and comprehensive information than single-modal data is currently a key research focus and difficulty.
[0003] In short, multimodal data fusion is to integrate data from different modalities and extract effective information from them, so that the final information obtained is more comprehensive and comprehensive, covering the information and characteristics provided by different modalities. However, this task is still full of challenges, mainly in the following aspects: (1) Differences between data modalities: The data forms, feature dimensions and data scales of various modalities vary greatly. For example, images are high-dimensional pixel matrices, texts are words or characters in sequence form, and sensor data are numerical data in time series. Therefore, how to effectively unify the model of data from different modalities to avoid information loss or error introduction has become a core issue. (2) Heterogeneous data representation: Data from different modalities not only differ in structure, but also may have uneven information density. For example, key information in speech may be sparse, while information in images may be denser. How to deal with these differences during fusion to avoid the model being biased towards one modality is an important challenge in designing multimodal data fusion methods. (3) Introduction of new modalities: In practical applications, new data modalities may be encountered. How to ensure model adaptation is also a major difficulty of traditional data fusion methods.
[0004] Currently, there are many achievements in the research of multimodal data fusion, and many methods attempt to address the above challenges from different angles. A common and simple approach is to match features and matching strategies based on manual design. Such methods usually rely on domain experts to manually design features (for example, extracting image features based on methods such as SIFT and HOG), and then fuse the data of different modalities through some weighting, matching or alignment methods. Features designed through expert experience can, to a certain extent, ensure the comparability of data from each modality and avoid information loss. With the development of machine learning, neural networks and deep learning have also been used in the research of multimodal data fusion, automatically extracting effective features from raw data of different modalities. However, if the same network model is used to fuse multimodal data, the effect is not good. If multiple networks are used to process different modal data separately, the computing power resources are too high, and the generalization ability of the model is weak when encountering new modal data and small sample scenarios. The graph convolutional network (GCN) based on the attention mechanism has solved the above problems to a certain extent. Through the attention mechanism, the model can be dynamically adjusted according to the relative importance of different modal input data, thereby realizing a network to solve the problem of feature extraction and fusion of multimodal data. However, the training process is very complicated and its accuracy cannot be guaranteed. Summary of the Invention
[0005] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the first aspect of the present application proposes a multimodal data fusion method based on meta-learning, including: taking the modal embedding layer as the bottom layer, the multimodal sharing layer as the middle layer, and the image processing model, text processing model and language processing model respectively connected to the multimodal sharing layer as the data processing layer to construct a cross-modal shared bottom layer encoder; using a cross-modal annotated dataset to train the cross-modal shared bottom layer encoder, and optimizing the initial parameters of the cross-modal shared bottom layer encoder through a model-independent meta-learning algorithm to obtain an initialization model; using a mixed cross-modal annotated dataset and an obtained semi-supervised pseudo-label dataset to train the initialization model, and updating the parameters of the initialization model based on a meta-learning transfer algorithm to obtain a transfer model; freezing the shared parameters of the transfer model, and using a small annotated sample to train the modal embedding layer of the transfer model to obtain a transfer model with rapid task adaptation; processing multimodal input data based on the transfer model with rapid task adaptation to obtain multimodal fusion data.
[0006] Optionally, before using the cross-modal annotation dataset to train the cross-modal shared underlying encoder, the method includes: reshaping the RGB image through cubic interpolation and then normalizing it to obtain a normalized image, dividing the normalized image into multiple patches, and flattening each patch into a multi-dimensional vector to obtain preprocessed image data; dividing the monophonic speech waveform into a short-time frame sequence with frames of a predetermined length and a frame shift of a predetermined length, calculating the spectrum of the short-time frame sequence through FFT, and mapping the spectrum to an 80-dimensional Mel frequency band to generate a Mel spectrum graph to obtain preprocessed text data; removing punctuation marks in the natural language text through regular expressions, retaining letters, numbers and Chinese characters, to obtain new natural language text; using the THULAC word segmentation tool to split the new natural language text into subwords; mapping the subwords to index sequences through the BERT word segmenter, and generating an input tensor based on the index sequence and the position encoding of the index sequence to obtain preprocessed speech data; determining the cross-modal annotation dataset based on the preprocessed image data, preprocessed text data and preprocessed speech data.
[0007] Optionally, before obtaining the semi-supervised pseudo-label data, the method further includes: inputting unlabeled data into an initialization model to obtain pseudo-label data, screening the pseudo-label data, and obtaining a semi-supervised pseudo-label data set.
[0008] Optionally, the cross-modal shared underlying encoder is trained using a cross-modal annotated dataset, and the initial parameters of the cross-modal shared underlying encoder are optimized through a model-independent meta-learning algorithm to obtain an initialized model, including: freezing the parameters of the modal embedding layer and the multimodal sharing layer; training the ViT model, BERT model and HuBERT model based on the cross-modal annotated dataset to obtain an image processing model, a text processing model and a language processing model respectively; constructing a meta-task set, training the data processing layer determined according to the image processing model, the text processing model and the language processing model based on the meta-task set, and performing gradient optimization on the data processing layer based on the meta-learning fast adaptation mechanism to obtain an initialized model.
[0009] Optionally, before training the ViT model, the BERT model, and the HuBERT model based on the cross-modal annotation dataset, the method further includes: freezing the parameters of the first 12 Transformer layers of the ViT model and opening the parameters of the last two Transformer layers; freezing the parameters of the first 8 Transformer layers and the word embedding layer of the BERT model, and opening the parameters of the last four Transformer layers; freezing the parameters of the bottom convolutional layer of the HuBERT model, and opening the parameters of the second to fourth Transformer layers.
[0010] Optionally, the meta-learning transfer algorithm is based on updating the parameters of the initialization model to obtain the transfer model, including: when performing inner-loop training on the initialization model, using the data processing layer to extract features of the training data, and embedding the features into the training data to obtain new training data; using the multimodal shared layer to process the new training data to obtain output data, and training the initialization model based on the first loss function, and using the cross-entropy loss function to update the task head parameters to obtain the initialization model after preliminary training; when performing outer-loop training on the initialization model, calculating the global loss function of the initialization model after preliminary training, updating the shared parameters based on the global loss function, and obtaining a trained transfer model.
[0011] Optionally, the multimodal shared layer includes 12 Transformer layers, each Transformer layer includes 12 self-attention heads and a feedforward neural network connected to the 12 self-attention heads.
[0012] Optionally, the set of constructed meta-tasks includes: a cross-modal alignment task, where the input modalities are image and text, and the output target is the matching result of the image and text; a label generation task, where the input modality is image, and the output target is text label; and a classification prediction task, where the input modalities are text and speech, and the output target is category label.
[0013] Optionally, the modality embedding layer includes a fully connected layer, a Softmax layer, a BiLSTM layer, a CRF layer and a Transformer decoder connected in sequence.
[0014] In order to solve the above problems, the second aspect of the present application provides a multimodal data fusion device based on meta-learning, including: an encoder construction module, which is used to use the modal embedding layer as the bottom layer, the multimodal sharing layer as the middle layer, and the image processing model, text processing model and language processing model respectively connected to the multimodal sharing layer as the data processing layer to construct a cross-modal shared bottom layer encoder; a model training module, which is used to train the cross-modal shared bottom layer encoder using a cross-modal annotation dataset, and optimize the initial parameters of the cross-modal shared bottom layer encoder through a model-independent meta-learning algorithm to obtain an initialized model; the initialization model is trained using a mixed cross-modal annotation dataset and the obtained semi-supervised pseudo-label dataset, and the parameters of the initialization model are updated based on the meta-learning transfer algorithm to obtain a transfer model; the shared parameters of the transfer model are frozen, and the modal embedding layer of the transfer model is trained using a small annotated sample to obtain a transfer model with rapid task adaptation; a fusion module, which is used to process multimodal input data based on the transfer model with rapid task adaptation to obtain multimodal fusion data.
[0015] The embodiment of the present application provides a multimodal data fusion method and device based on meta-learning. Compared with the existing technology, the beneficial effects of the method are as follows: by taking the modal embedding layer as the bottom layer, the multimodal sharing layer as the middle layer, and the image processing model, text processing model and language processing model respectively connected to the multimodal sharing layer as the data processing layer, a cross-modal shared bottom layer encoder is constructed, which can effectively extract heterogeneous data features of different types (such as images, text, audio, etc.) and reduce the dependence on a large amount of labeled data; the cross-modal shared bottom layer encoder is trained using a cross-modal labeled dataset, and the cross-modal shared bottom layer encoder is trained through a model-independent meta-learning algorithm. The method optimizes the initial parameters of the cross-modal shared underlying encoder to obtain the initialization model; uses the mixed cross-modal annotated dataset and the obtained semi-supervised pseudo-label dataset to train the initialization model, and updates the parameters of the initialization model based on the meta-learning migration algorithm to obtain the migration model; freezes the shared parameters of the migration model, and uses small annotated samples to train the modal embedding layer of the migration model to obtain a migration model with rapid task adaptation, which solves the problems of excessive model computing resource consumption, long training time, and low efficiency in multimodal data fusion; processes multimodal input data based on the migration model with rapid task adaptation to obtain multimodal fusion data. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] To more clearly illustrate the technical solution of this application, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.
[0017] Figure 1 A flowchart of a multimodal data fusion method based on meta-learning provided in an embodiment of the present application;
[0018] Figure 2 A schematic diagram of the meta-learning process of a multimodal data fusion method based on meta-learning provided in an embodiment of the present application. DETAILED DESCRIPTION
[0019] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0020] This specification provides method operation steps as described in the embodiments or flowcharts, but more or fewer operation steps may be included based on routine or non-creative work. When implemented in an actual system or server product, the methods shown in the embodiments or figures may be executed sequentially or in parallel (for example, in a parallel processor or multi-threaded processing environment).
[0021] The purpose of this application is to build a cross-modal unified modeling system based on a meta-learning framework to address the core pain points of existing multimodal data fusion technology, such as the lack of adaptability caused by the reliance on expert experience of artificial features, the high computational cost caused by the redundancy of deep learning model parameters, and the weak generalization ability of traditional methods for new modalities / small sample scenarios. Specifically, this application uses a meta-learning transfer strategy to achieve normalized training of basic models in multimodal tasks (long and short text element extraction, image OCR recognition, audio transcription, etc.), avoiding the traditional method's reliance on artificial feature design, and automatically mining the deep correlation of cross-modal data; designing a lightweight architecture with parameter sharing (Transformer encoder) to replace the traditional multi-branch network's repeated training mode to solve the resource bottleneck problem of deep learning; using the meta-learning fast adaptation mechanism (MAML algorithm), the model only needs a small amount of gradient updates (1-2 steps) to complete the adaptation when facing new modalities or small sample tasks, which improves training efficiency compared to traditional methods and significantly enhances the model's generalization ability for unknown scenarios; constructing a cross-modal semantic alignment network, through contrast loss and semi-supervised pseudo-labeling technology, forcing different modal features to be accurately aligned in a unified semantic space, solving the model bias problem caused by uneven representation of heterogeneous data, and improving the accuracy of multimodal joint tasks.
[0022] This application proposes a meta-learning-based multimodal data fusion method, which aims to solve several technical difficulties in current multimodal data processing, including how to effectively extract features of heterogeneous data of different types (such as images, text, audio, etc.) and how to reduce dependence on large amounts of labeled data.
[0023] The multimodal data fusion method based on meta-learning can include the following execution process:
[0024] S10. Use the modal embedding layer as the bottom layer, the multimodal sharing layer as the middle layer, and the image processing model, text processing model, and language processing model connected to the multimodal sharing layer as the data processing layer to build a cross-modal shared bottom layer encoder.
[0025] In one embodiment of the present application, before using the cross-modal annotation dataset to train the cross-modal shared underlying encoder, the method further includes the following execution process:
[0026] Specifically, the RGB image is reshaped and then normalized by using cubic interpolation to obtain a normalized image, the normalized image is divided into a plurality of patches, and each patch is flattened into a multi-dimensional vector to obtain preprocessed image data;
[0027] For example, the processor may use bicubic interpolation to scale an RGB image with a resolution of 640*480 to 224x224 pixels. The command that may be used in this process is Image_resized=Resize(Image_raw, (224, 224)).
[0028] The processor may then normalize the pixel value to [-1, 1] using the formula (Pixel_value-μ) / σ, where μ=[0.485, 0.456, 0.406] and σ=[0.229, 0.224, 0.225].
[0029] Finally, the processor divides the image into patches of 16x16 pixels using the command Pixel_norm=Block, generating a total of 14x14=196 patches, each of which is flattened into a 768-dimensional vector (3x16x16=768).
[0030] Through the above steps, the preprocessed image data can be obtained.
[0031] The monophonic speech waveform is divided into a short-time frame sequence with a predetermined frame length and a predetermined frame shift as units, the spectrum of the short-time frame sequence is calculated by FFT, and the spectrum is mapped to an 80-dimensional Mel frequency band to generate a Mel frequency spectrum graph to obtain the preprocessed text data;
[0032] Exemplarily, for speech data, when framing, the processor divides the input monophonic speech waveform signal into a short-time frame sequence with a frame length of 25ms and a frame shift of 10ms, where the framing command can be Frames=Split(Speech_wave, frame_len=400, fame_shift=160).
[0033] Afterwards, the processor can perform Mel spectrum calculation. For example, the processor can calculate the spectrum through 512-point FFT, map it to an 80-dimensional Mel frequency band, and generate a Mel spectrum graph with a shape of (T, 80) (T is the number of frames). In this way, the preprocessed text data can be obtained.
[0034] Punctuation marks in natural language text are removed through regular expressions, while retaining letters, numbers, and Chinese characters to obtain new natural language text; the new natural language text is split into subwords using the THULAC word segmentation tool; the subwords are mapped to index sequences through the BERT word segmenter, and input tensors are generated based on the index sequences and the positional encoding of the index sequences to obtain preprocessed speech data.
[0035] For example, for text data, the processor may remove punctuation marks (such as “,”) from natural language text (such as “The weather was sunny on April 26, 2005”) through regular expressions, and only retain letters, numbers, and Chinese characters.
[0036] The processor can then use the THULAC word segmentation tool to split the text into subwords, such as "2025 / April / 26 / weather / ".
[0037] Finally, the processor can map the subwords into index sequences through the BERT tokenizer and combine them with the positional encoding to generate an input tensor with a shape of (seq_len, 768).
[0038] Through the above operations, each modal data (such as image patches, text word vectors, and speech mel-spectrograms) is preprocessed to obtain preprocessed data. The preprocessed data is then added with exclusive labels ([IMG] / [TXT] / [AUD]) through the modal embedding layer and then input into the multimodal sharing layer to generate a common feature vector Z to achieve cross-modal semantic alignment (for example, the features of the image "car" and the text "car" are highly correlated in the Z space).
[0039] In one embodiment of the present application, the multimodal shared layer includes 12 Transformer layers, each Transformer layer includes 12 self-attention heads and a feedforward neural network connected to the 12 self-attention heads.
[0040] It is worth noting that this application uses a unified Transformer encoder as a multimodal shared layer (such as Figure 1 (as shown) replaces the traditional multi-branch independent model. After preprocessing, the image, text, and speech data are all passed through the encoder to generate a common feature vector Z. For example, for example, the image, text, and speech share the parameters θ of the same 12-layer Transformer. The traditional multi-branch model requires independent training of vision, text, and speech encoders, while the present application reduces repeated parameters by sharing encoders. The present application can also reduce computational costs, and the shared layer only needs to be calculated once, avoiding repeated forward propagation of multimodal branches in traditional methods.
[0041] In one embodiment of the present application, the modality embedding layer includes a fully connected layer, a Softmax layer, a BiLSTM layer, a CRF layer and a Transformer decoder connected in sequence.
[0042] Table 1 Composition list of modality embedding layer
[0043]
[0044] For example, Table 1 lists the components of the modality embedding layer. The modality embedding layer, also known as the task-specific header, adopts a lightweight design structure. Each task (such as classification or extraction) only adds a single fully connected layer as the task header, and the parameter size is less than 5% of the total model.
[0045] S20. Use the cross-modal annotated dataset to train the cross-modal shared underlying encoder, and optimize the initial parameters of the cross-modal shared underlying encoder through a model-independent meta-learning algorithm to obtain an initialization model; use the mixed cross-modal annotated dataset and the obtained semi-supervised pseudo-label dataset to train the initialization model, and update the parameters of the initialization model based on the meta-learning transfer algorithm to obtain a transfer model; freeze the shared parameters of the transfer model, and use the small annotated sample to train the modality embedding layer of the transfer model to obtain a transfer model that can quickly adapt to the task;
[0046] Meta-learning-based multimodal fusion technology is an emerging research direction in machine learning in recent years. By combining meta-learning and multimodal techniques, it aims to enable models to learn from small amounts of cross-modal data and quickly adapt to new tasks. The model is trained on multiple representative tasks that involve different multimodal data input and output structures.
[0047] Figure 2 This is a schematic diagram of the meta-learning process. Meta-learning-based multimodal fusion technology consists of the first and second stages. The first stage is single-modal pre-training (feature extraction layer reuse). Single-modal models include the ViT (Vision Transformer) image processing model, the BERT (Bidirectional Encoder Representations from Transformers) text processing model, and the HuBERT (Hidden Unit BERT) audio model.
[0048] Among them, the cross-modal shared underlying encoder is trained using a cross-modal annotated dataset, and the initial parameters of the cross-modal shared underlying encoder are optimized through a model-independent meta-learning algorithm. The initialization model can include the following execution process:
[0049] S201, freezing the parameters of the modal embedding layer and the multimodal sharing layer;
[0050] In one embodiment of the present application, the multimodal shared layer includes 12 Transformer layers, each Transformer layer includes 12 self-attention heads and a feedforward neural network connected to the 12 self-attention heads.
[0051] For example, by reusing the "general feature extraction capability" of the pre-trained model, the processor can avoid training a high-complexity model from scratch and reduce the basic feature learning cost by more than 90%.
[0052] S202: Training a ViT model, a BERT model, and a HuBERT model based on a cross-modal annotation dataset to obtain an image processing model, a text processing model, and a language processing model, respectively.
[0053] In one embodiment of the present application, before training the ViT model, the BERT model, and the HuBERT model based on the cross-modal annotation dataset, the method further includes the following execution process:
[0054] Freeze the parameters of the first 12 Transformer layers of the ViT model, open the parameters of the last two Transformer layers, and train the ViT model based on the cross-modal annotated dataset;
[0055] Freeze the parameters of the first eight Transformer layers and word embedding layers of the BERT model, and open the parameters of the last four Transformer layers;
[0056] Freeze the parameters of the underlying convolutional layers of the HuBERT model and open the parameters of the second to fourth Transformer layers.
[0057] For example, during the training of the initialization model, for the vision (ViT) model, the processor can freeze the first 12 layers of Transformer, open the last 2 layers, and output 768-dimensional visual features. For the text (BERT) model, the processor can freeze the word embedding layer and the first 8 layers of Transformer, open the last 4 layers, and output 768-dimensional text features. For the speech (HuBERT) model, the processor can freeze the bottom convolutional layer, open the top 4 layers of Transformer, output 1024-dimensional speech features, and compress them to 768 dimensions through the linear layer. This can reduce the parameter size by 40% to reduce the complexity of the training process.
[0058] S203. Construct a meta-task set, and train a data processing layer determined according to the image processing model, the text processing model, and the language processing model based on the meta-task set; perform gradient optimization on the data processing layer based on a meta-learning fast adaptation mechanism to obtain an initialized model.
[0059] Among them, the meta-task set constructed includes cross-modal alignment tasks, where the input modalities are images and texts, and the output targets are the matching results of images and texts; label generation tasks, where the input modalities are images, and the output targets are text labels; and classification prediction tasks, where the input modalities are text and speech, and the output targets are category labels.
[0060] For example, the constructed meta-task set can be constructed using the following list:
[0061] Table 2 Task set construction list
[0062]
[0063]
[0064] For example, first, the processor uses 10 2 -10 3 The processor uses cross-modal contrastive loss (for example, enforcing the feature vector distance between "car image" and "car text" to be less than 0.2) to train the model using 100 car photos and text descriptions, forming a basic framework (initialized model). However, the model can only recognize the few trained samples.
[0065] In an embodiment of the present application, step S20 may also include the following execution process:
[0066] S204, when performing inner loop training on the initialized model, extracting features of the training data using the data processing layer, and embedding the features into the training data to obtain new training data;
[0067] S205. Process the new training data using the multimodal shared layer to obtain output data, train the initialization model based on the first loss function, and update the task head parameters using the cross entropy loss function to obtain the initialization model after preliminary training;
[0068] S206. When performing outer loop training on the initialized model, calculate the global loss function of the initialized model after preliminary training, update the shared parameters based on the global loss function, and obtain a trained migration model.
[0069] The gradient optimization process of the data processing layer based on the meta-learning fast adaptation mechanism includes gradient optimization driven by the MAML algorithm. Specifically, in the inner loop, the processor quickly adjusts the task-specific parameters (classification head weights) for each task (such as image classification) using a small amount of data (such as 5 car images), with the goal of minimizing the training loss (cross-entropy loss) of the current task. In the outer loop (cross-task generalization), the processor collects the test loss of all tasks and reversely optimizes the shared parameter θ to make the model perform evenly across all tasks, forming a "universal learning ability."
[0070] It is worth noting that the above process is the second stage of the meta-learning training mechanism of this application. In this second stage, the processor can accelerate convergence, reduce the number of iterations, and obtain a rapidly adaptable inner and outer loop architecture. Through task-level optimization of the inner loop, each task only requires 1-2 gradient updates (traditional methods require hundreds of steps), quickly adjust task head parameters, and reduce the training time of a single task.
[0071] During the actual execution process, when the processor performs inner loop training on the initialized model, the processor only updates the task head parameters, and the shared encoder parameters remain unchanged. This reduces the amount of computation for each parameter update by more than 95%, and when adapting to new tasks, only the gradient of the task head (about 100,000 parameters) needs to be calculated, rather than the billions of parameters of the entire model. During the meta-level optimization process, the processor updates the shared parameters by summing the cross-task losses. Each round of training covers the common features of multiple tasks, avoiding falling into the local optimum of a single task. The beneficial effect of this is that data can be used efficiently, and it can also reduce annotation costs and the amount of training data.
[0072] For example, in the outer loop, the processor can reversely update the shared parameter θ based on the test loss of all tasks, so that the model can learn cross-task commonalities (such as "object contours" and "keyword semantics"). It can be understood that the model is like a "student", using "real test questions" (labeled data) and "simulated test questions" (pseudo-labeled data) to practice different subjects (multi-tasks), and summarizes general problem-solving methods (such as "finding key features" and "associated information") through "brushing questions". These methods are stored in the "brain" (shared parameter θ). The processor can use "general capabilities" to solve "new problems". When facing new tasks, only 20-50 labeled samples are needed.
[0073] Through multi-task training, the processor uses the model's shared parameters θ to capture the following common features: cross-modal alignment capability, forming a semantic association between the "circle" in the image, the "wheel" in the text, and the "rolling" in the speech. Feature abstraction capability, extracting the common feature "vehicle attributes" from "car images", "car text", and "car speech". The final effect is that when faced with a new task, the model only needs 1-2 steps of gradient updates (such as adjusting the classification head) to reuse the common features in θ and achieve "fast cold start". The above process enables the model to learn common features across tasks and improve its generalization ability on new tasks. When the model encounters a new task, it can adapt quickly and only requires a small amount of example data to optimize the model parameters through gradient updates. This rapid adaptability makes meta-learning extremely flexible and efficient in practical applications.
[0074] In one embodiment of the present application, before using the mixed cross-modal annotated dataset and the obtained semi-supervised pseudo-labeled dataset to train the initialization model, the method further includes:
[0075] The unlabeled data is input into the initialization model to obtain pseudo-labeled data, and the pseudo-labeled data is filtered to obtain a semi-supervised pseudo-labeled dataset.
[0076] Specifically, the processor can process the initial model with 10 4 -10 5 Generate predicted labels based on the unlabeled data (such as outputting a probability of "car" of 0.85 for an unlabeled image → pseudo label "vehicle"). When screening pseudo labels, the processor can screen reliable pseudo labels by setting a confidence threshold (such as >0.7), and can also filter noise through cross-modal consistency verification (such as cosine similarity of image and text features >0.9). For the pseudo labels after filtering the noise, they are manually inspected. For example, 1,000 labels are randomly inspected and 50 errors are corrected to ensure that the accuracy of the pseudo labels is >85%. Finally, the processor can compare the model to an "intern" and use the learned initialization model to "guess labels" for 100,000 unlabeled images. The ones that are guessed correctly (such as "four wheels + can run → car") are kept, and the ones that are not guessed correctly (such as "airplane → car") are thrown away. Finally, the "tutor" (manual) is asked to check and correct them to obtain a high-quality "mock test paper".
[0077] For example, the semi-supervised pseudo-label generation process can be a process in which the processor trains an initial model for 10 4 -10 5 Generate high-confidence pseudo labels (probability > 0.8) from unlabeled data. Manually check 5% of the pseudo labels (e.g., correct 50 errors in 1000 samples) to ensure accuracy > 85%. This has the beneficial effect of reducing the labeling cost by 90%, while in traditional fully supervised training, it takes about 10 4Level 103 labeled samples are required, while this application only requires 103 levels of real labeled data to start. In this application, the training data volume remains unchanged but efficiency is improved. Moreover, the pseudo-labeled data adds semantic diversity, which improves the model's learning effect by 15% with the same amount of data. At the same time, this application can reduce the annotation cost and improve the model's adaptability and generalization ability when facing new tasks or new modalities.
[0078] It should be noted that the above two stages of training solve the problems of excessive consumption of model computing resources, long training time, and low efficiency in multimodal data fusion.
[0079] S30, based on the task rapid adaptation migration model to process multimodal input data and obtain multimodal fusion data
[0080] The "learning-shared model" is a combination of meta-learning and parameter sharing. Its core is to achieve efficient modeling through cross-modal and cross-task parameter sharing and meta-knowledge learning, achieving "one-time training, multiple-scenario reuse" and fundamentally addressing the high cost and low efficiency issues of traditional multimodal models. (The initialization of the model) allows the model to quickly adapt to new tasks using a small amount of labeled data, significantly reducing the demand for labeled data.
[0081] Quickly adapt to new tasks and data distributions. Through meta-learning technology, this application enables the model to learn initial model parameters that are universally adaptable to different tasks during training. When faced with new tasks, the model can quickly adapt and make fine-tuning.
[0082] Cross-modal learning ability Meta-learning enables the model to establish effective connections between different modalities, improve cross-modal learning ability, complement each other with features of different modalities, and solve the deviation or noise problems that may occur in a single modality.
[0083] Improve the generalization ability and robustness of the model through the optimization of the meta-learning framework, and have better initialization parameters for different tasks, avoiding the long training time required for large-scale training in traditional deep learning, and can still ensure a high accuracy when the sample size is low.
[0084] Test data
[0085] 3,000 images, 3,000 voices, and 4,000 text data.
[0086] Table 3 Performance comparison of meta-learning and other methods in processing multimodal data
[0087] Meta-learning Federated Learning Generative Adversarial Networks Multimodal data recognition accuracy 78％ 75％ 76％ Adaptability high Poor generally Sample demand Low high high Resource consumption Low generally generally
[0088] Table 4 Image data recognition accuracy based on meta-learning framework
[0089] Experimental Objectives Whether the recognition accuracy of image data meets the standards Index requirements Recognition accuracy reaches 70% Test set A total of 3,000 images Test results Semantic understanding accuracy reaches 72% Whether it meets the standards meet the standards
[0090] Table 6 Speech data recognition accuracy based on the meta-learning framework
[0091] Experimental Objectives Whether the recognition accuracy of voice data meets the standard Index requirements The recognition detection accuracy rate reaches 70% Test set A total of 3,000 voice data Test results Semantic understanding accuracy reaches 71% Whether it meets the standards meet the standards
[0092] The above results prove that the multimodal data fusion method based on meta-learning has excellent performance, the best multimodal data recognition accuracy, high adaptability, low sample requirements, and low resource consumption.
[0093] In summary, this application uses meta-learning multimodal data fusion technology to effectively extract and integrate information from different modalities, extract valuable information from different modalities, overcome the limitations of single modality data, and significantly improve the generalization ability of tasks.
[0094] Based on the above embodiments, the present application also provides a multimodal data fusion device based on meta-learning, including an encoder construction module, a model training module and a fusion module, wherein the encoder construction module is used to use the modal embedding layer as the bottom layer, the multimodal sharing layer as the middle layer, and the image processing model, text processing model and language processing model respectively connected to the multimodal sharing layer as the data processing layer to construct a cross-modal shared bottom layer encoder; the model training module is used to train the cross-modal shared bottom layer encoder using a cross-modal annotation dataset, and optimize the initial parameters of the cross-modal shared bottom layer encoder through a model-independent meta-learning algorithm to obtain an initialization model; the initialization model is trained using a mixed cross-modal annotation dataset and the obtained semi-supervised pseudo-label dataset, and the parameters of the initialization model are updated based on the meta-learning transfer algorithm to obtain a transfer model; the shared parameters of the transfer model are frozen, and the modal embedding layer of the transfer model is trained using a small annotated sample to obtain a transfer model with rapid task adaptation; the fusion module is used to process multimodal input data based on the transfer model with rapid task adaptation to obtain multimodal fusion data.
[0095] In another embodiment provided in the present application, a device is also provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the meta-learning-based multimodal data fusion method in the embodiment of the present application.
[0096] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which at least one instruction, at least one program, code set or instruction set is stored. The at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the multimodal data fusion method based on meta-learning in the embodiment of the present application.
[0097] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes a plurality of computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes multiple available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)).
[0098] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.
[0099] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are described briefly because they are generally similar to the method embodiments. For related portions, reference can be made to the description of the method embodiments.
[0100] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the scope of protection of the present application.
Claims
1. A multimodal data fusion method based on meta-learning, characterized in that: include: The modality embedding layer is used as the bottom layer, the multimodal sharing layer is used as the middle layer, and the image processing model, text processing model, and language processing model connected to the multimodal sharing layer are used as the data processing layer to build a cross-modal shared bottom layer encoder. A cross-modal shared underlying encoder is trained using a cross-modal annotated dataset, and the initial parameters of the cross-modal shared underlying encoder are optimized using a model-independent meta-learning algorithm to obtain an initialization model. The initialization model is trained using a mixed cross-modal annotated dataset and the obtained semi-supervised pseudo-label dataset, and the parameters of the initialization model are updated based on a meta-learning transfer algorithm to obtain a transfer model. The shared parameters of the transfer model are frozen, and the modality embedding layer of the transfer model is trained using a small number of annotated samples to obtain a transfer model that can quickly adapt to tasks. The migration model based on rapid task adaptation processes multimodal input data to obtain multimodal fusion data.
2. The multimodal data fusion method based on meta-learning according to claim 1, characterized in that: Before training the cross-modal shared underlying encoder using the cross-modal annotated dataset, the method includes: The RGB image is reshaped and normalized by cubic interpolation to obtain a normalized image, the normalized image is divided into multiple patches, and each patch is flattened into a multi-dimensional vector to obtain preprocessed image data; The monophonic speech waveform is divided into a short-time frame sequence by a frame of a predetermined length and a frame shift of a predetermined length. The spectrum of the short-time frame sequence is calculated by FFT, and the spectrum is mapped to an 80-dimensional Mel frequency band to generate a Mel frequency spectrum graph to obtain the preprocessed text data. Use regular expressions to remove punctuation from natural language text, retaining letters, numbers, and Chinese characters to generate new natural language text; use the THULAC word segmentation tool to split the new natural language text into subwords; use the BERT word segmenter to map the subwords to index sequences, and generate input tensors based on the index sequences and their positional encodings to obtain preprocessed speech data; A cross-modal annotation dataset is determined based on the preprocessed image data, the preprocessed text data, and the preprocessed speech data.
3. The multimodal data fusion method based on meta-learning according to claim 1, characterized in that: Before using the mixed cross-modal annotated dataset and the obtained semi-supervised pseudo-labeled dataset to train the initialization model, the method further includes: The unlabeled data is input into the initialization model to obtain pseudo-labeled data, and the pseudo-labeled data is filtered to obtain a semi-supervised pseudo-labeled dataset.
4. The multimodal data fusion method based on meta-learning according to claim 1, characterized in that: The cross-modal shared underlying encoder is trained using a cross-modal annotated dataset, and the initial parameters of the cross-modal shared underlying encoder are optimized through a model-independent meta-learning algorithm to obtain an initialized model, including: Freeze the parameters of the modality embedding layer and the multimodal sharing layer; Based on the cross-modal annotation dataset, the ViT model, BERT model, and HuBERT model are trained to obtain image processing models, text processing models, and language processing models respectively; Construct a meta-task set, and train the data processing layer determined by the image processing model, text processing model, and language processing model based on the meta-task set; perform gradient optimization on the data processing layer based on the meta-learning fast adaptation mechanism to obtain an initialized model.
5. The multimodal data fusion method based on meta-learning according to claim 1, characterized in that: Before training the ViT model, the BERT model, and the HuBERT model based on the cross-modal annotation dataset, the method further includes: Freeze the parameters of the first 12 Transformer layers of the ViT model and open the parameters of the last two Transformer layers; Freeze the parameters of the first eight Transformer layers and word embedding layers of the BERT model, and open the parameters of the last four Transformer layers; Freeze the parameters of the underlying convolutional layers of the HuBERT model and open the parameters of the second to fourth Transformer layers.
6. The multimodal data fusion method based on meta-learning according to claim 1, characterized in that: The meta-learning transfer algorithm is used to update the parameters of the initialization model to obtain the transfer model, including: When performing inner loop training on the initialized model, the data processing layer is used to extract the features of the training data, and the features are embedded into the training data to obtain new training data; Use the multimodal shared layer to process the new training data to obtain the output data, train the initialization model based on the first loss function, and use the cross entropy loss function to update the task head parameters to obtain the initialization model after preliminary training; When the initialization model is trained in the outer loop, the global loss function of the initialization model after preliminary training is calculated, and the shared parameters are updated based on the global loss function to obtain the trained migration model.
7. The multimodal data fusion method based on meta-learning according to claim 1, characterized in that: The multimodal shared layer includes 12 Transformer layers, each Transformer layer includes 12 self-attention heads and a feedforward neural network connected to the 12 self-attention heads.
8. The multimodal data fusion method based on meta-learning according to claim 4, characterized in that: The construction meta-task set includes: Cross-modal alignment task, the input modalities are image and text, and the output target is the matching result of image and text; Label generation task, the input modality is an image and the output target is a text label; Classification prediction tasks, the input modalities are text and speech, and the output target is the category label.
9. The multimodal data fusion method based on meta-learning according to claim 1, characterized in that: The modality embedding layer includes a fully connected layer, a Softmax layer, a BiLSTM layer, a CRF layer and a Transformer decoder, which are connected in sequence.
10. A multimodal data fusion device based on meta-learning, characterized in that: include: The encoder construction module is used to construct a cross-modal shared bottom-layer encoder using the modal embedding layer as the bottom layer, the multimodal sharing layer as the middle layer, and the image processing model, text processing model, and language processing model connected to the multimodal sharing layer as the data processing layer; The model training module is used to train the cross-modal shared underlying encoder using a cross-modal annotated dataset and optimize the initial parameters of the cross-modal shared underlying encoder through a model-independent meta-learning algorithm to obtain an initialized model. The initialized model is trained using a mixed cross-modal annotated dataset and the obtained semi-supervised pseudo-label dataset, and the parameters of the initialized model are updated based on a meta-learning transfer algorithm to obtain a transferred model. The shared parameters of the transferred model are frozen, and the modality embedding layer of the transferred model is trained using a small number of annotated samples to obtain a transferred model that can be quickly adapted to the task. The fusion module is used to process multimodal input data based on the migration model with rapid task adaptation to obtain multimodal fusion data.