Multi-modal data fusion method and system for large model training

Abstract

The invention discloses a multi-modal data fusion method for large model training, which belongs to the technical field of large model training, and comprises the following steps: extracting detail features of text data and image data through a pre-trained semantic feature extraction model to obtain text feature data and image feature data; the method comprises the following steps: generating supplementary text feature data for text features by using image data by using a collaborative self-variation encoder, carrying out weighted fusion on the text feature data by using the supplementary text feature data, and fusing local image feature data and global image feature data to form pre-fused enhanced features; according to the method, full detail features can be fully utilized by subsequent fusion data, image and text features are fused through interactive query and semantic enhancement, and parameters of a large model are updated by utilizing rich fusion semantic information in large model training, so that the image-text recognition capability of the large model is improved.

Description

Technical Field

[0001] This invention belongs to the field of large model training technology, specifically relating to a multimodal data fusion method and system for large model training. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, large-scale models that integrate multimodal information such as text, images, and speech have become a key driving force for the next generation of AI. These models have shown great potential in areas such as image description generation, visual question answering, and multimodal content retrieval, their core being their ability to understand and associate deep semantics between different modalities. Currently, mainstream multimodal fusion methods mainly rely on pre-trained models (such as BERT, ViT, and ResNet) to extract single-modal features separately, and then perform feature interaction through simple concatenation, addition, or attention-based weighted summation. Although these methods have achieved some success, they still have significant limitations when dealing with complex tasks requiring fine-grained alignment and deep semantic understanding. For example, in text-to-image generation tasks, the detailed texture and spatial structure of the generated image are often blurred and distorted; in multimodal named entity recognition tasks, the correspondence between text descriptions and image regions is not fully explored and utilized, leading to biases in entity type judgment. The essence of these problems lies in the fact that existing methods are still at a relatively shallow stage in representing and fusing multimodal data, failing to truly achieve semantic unification and synergy between modalities.

[0003] As mentioned above, this paper presents a multimodal data fusion method and system for training large models that can perform bidirectional fusion processing of detailed features of text and images, and improve the image and text recognition capabilities of large models during training. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal data fusion method for training large models, in order to solve the above-mentioned problems existing in the prior art.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a multimodal data fusion method for training large models, comprising: Text data and image data are acquired. The text data is then processed by a pre-trained semantic feature extraction model to obtain text feature data. The image data is then processed by the same semantic feature extraction model to obtain image feature data. The image feature data includes local image feature data and global image feature data. Based on the global image feature data, supplementary text feature data is generated using a collaborative autovariable encoder. The text feature data and the supplementary text feature data are then weighted and fused to obtain enhanced text feature data. The local image feature data and the global image feature data are then weighted and fused to obtain enhanced image feature data. By utilizing the cross-attention mechanism, cross-querying and semantic enhancement are performed on the enhanced text feature data and the enhanced image feature data to complete bidirectional interactive fusion and obtain image-text fusion feature data; The image-text fusion feature data is used as the output of the large model parameter update to train the large model.

[0006] In one possible design, the pre-training method for the semantic feature extraction model includes: The original text data is obtained from the ultra-large-scale dataset as the original pre-training text sample. The original pre-training text sample is then masked according to a first preset ratio to obtain the pre-training text sample. The BERT model is selected as the base model. The pre-trained text samples are used as input and the original text data is used as output. The BERT model is pre-trained based on self-supervised learning to obtain a text semantic feature extraction sub-model. Obtain raw image data from a massive dataset and obtain the real labels corresponding to the raw image data. The real labels of the raw image data are used to represent the real image category information and complete semantic information corresponding to the raw image data. The original image data is used as the first pre-trained image sample, and the true label of the original image data is used as the first pre-trained image semantic feature. The ResNet model is selected as the base model. The first pre-trained image sample is used as the input and the semantic features of the first pre-trained image are used as the output, forming a one-to-one input and output pair. Based on supervised learning, the ResNet model is pre-trained to obtain the first image semantic feature extraction sub-model. The original image data is processed into image blocks to obtain multiple image blocks. Each image block is used as an original second pre-training image sample. The original second pre-training image sample is masked according to a second preset ratio to obtain a second pre-training image sample. The ViT model is selected as the base model. The second pre-trained image sample is used as the input and the real label of the original image data is used as the output. The ViT model is pre-trained based on self-supervised learning to obtain the second image semantic feature extraction sub-model. The text semantic feature extraction sub-model is used as the text feature extraction encoder, the first image semantic feature extraction sub-model is used as the local image feature extraction encoder, and the second image semantic feature extraction sub-model is used as the global image feature extraction encoder. Based on the text feature extraction encoder, the local image feature extraction encoder, and the global image feature extraction encoder, a semantic feature extraction model is formed.

[0007] In one possible design, the semantic feature extraction model is a multi-head model, including a text input head and an image input head, as well as a text feature output head, a local image feature output head, and a global image feature output head; Accordingly, acquiring text data and image data, extracting features from the text data using a pre-trained semantic feature extraction model to obtain text feature data, and extracting features from the image data using the same semantic feature extraction model to obtain image feature data, includes: Extracting interconnected text and image data from massive datasets; The text data is cleaned and standardized. The processed text data is then input into the semantic feature extraction model through the text input head. The text feature extraction encoder is used to extract features from the text data. The text feature output head outputs text feature data, which includes word vectors and sentence vectors. The image data is subjected to size unification and normalization processing. The processed image data is then input into the semantic feature extraction model through the image input head. The local image feature extraction encoder and the global image feature extraction encoder are used to extract features from the image data. Local image feature data is output through the local image feature output head, and global image feature data is output through the global image feature output head. The local image feature data includes feature maps, and the global image feature data includes image semantic vectors.

[0008] In one possible design, supplementary text feature data is generated based on the global image feature data using a cooperative autovariable encoder, including: The global image feature data is used as high-dimensional global image features. The high-dimensional global image features are then reduced in dimensionality using the convolutional layer and average pooling layer of the cooperative autovariable encoder to obtain the dimensionality-reduced and compressed low-dimensional global image features. The low-dimensional global image features are encoded using the low-dimensional coding network of the cooperative autovariable encoder to obtain a Gaussian distribution of low-dimensional image features, wherein the Gaussian distribution of low-dimensional image features is defined by the mean and logarithmic variance of the low-dimensional image features. The text feature data is input into the convolutional layer of the collaborative autovariable encoder to obtain high-dimensional text features; The high-dimensional global image features and the high-dimensional text features are concatenated to obtain high-dimensional image-text fusion features; The low-dimensional image feature Gaussian distribution is input into the joint high-dimensional coding network of the collaborative autovariable encoder to jointly encode the low-dimensional image feature Gaussian distribution and the high-dimensional image-text fusion feature to obtain the high-dimensional image-text fusion feature Gaussian distribution, wherein the high-dimensional image-text fusion feature Gaussian distribution is defined by the mean and logarithmic variance of the high-dimensional image-text fusion feature. Using reparameterized sampling, latent variables are sampled from the Gaussian distribution of the low-dimensional image features and the Gaussian distribution of the high-dimensional image-text fusion features, respectively, to obtain the latent variables of the low-dimensional image features and the latent variables of the high-dimensional image-text fusion features. Based on the preset high-dimensional variable weighting coefficients in the collaborative autovariable encoder, the low-dimensional image feature latent variable and the high-dimensional image-text fusion feature latent variable are weighted and summed to obtain the joint latent variable of high- and low-dimensional fusion features. The high- and low-dimensional fused features are decoded together with the latent variables to generate high- and low-dimensional fused features, which are then used as supplementary text feature data.

[0009] In one possible design, the text feature data and the supplementary text feature data are weighted and fused to obtain enhanced text feature data, including: Obtain a preset single-modal fusion model, wherein the single-modal fusion model includes a text feature processing convolutional block and an image feature processing convolutional block, wherein the text feature processing convolutional block is used to process text feature data and supplement text feature data, and the image feature processing convolutional block is used to process local image feature data and global image feature data; Using the text feature processing convolutional block in the unimodal fusion model, semantic features are extracted from the text feature data and the supplementary text feature data to obtain text semantic feature data and supplementary text semantic feature data; Obtain the preset text feature weight matrix in the single-modal fusion model, wherein the text feature weight matrix includes the preset weights assigned to the text semantic feature data and the supplementary text semantic feature data in the text feature processing convolution block; Based on the text feature weight matrix, the text semantic feature data and the supplementary text semantic feature data are weighted and summed to obtain the enhanced text feature data. Accordingly, weighted fusion of the local image feature data and the global image feature data to obtain enhanced image feature data includes: Using the image feature processing convolutional block in the single-modal fusion model, semantic features are extracted from the local image feature data and the global image feature data to obtain local image semantic feature data and global image semantic feature data. Obtain the preset image feature weight matrix in the single-modal fusion model, wherein the image feature weight matrix includes the preset weights assigned to the local image semantic feature data and the global image semantic feature data in the image feature processing convolution block; Based on the image feature weight matrix, the local image semantic feature data and the global image semantic feature data are weighted and summed to obtain enhanced image feature data.

[0010] In one possible design, a cross-attention mechanism is used to perform cross-querying and semantic enhancement on the enhanced text feature data and the enhanced image feature data to complete bidirectional interactive fusion and obtain image-text fusion feature data, including: The enhanced text feature data is used as a query vector to query the enhanced image feature data to obtain the first query result of the enhanced text feature data on the enhanced image feature data, and the similarity score between the image and the text corresponding to the first query result is calculated. Based on the image-to-text similarity score corresponding to the first query result, the enhanced image feature data is used to semantically enhance the enhanced text feature data to obtain image-enhanced text feature data. The enhanced image feature data is used as a query vector to query the enhanced text feature data, so as to obtain the second query result of the enhanced image feature data on the enhanced text feature data, and calculate the similarity score between the text and the image corresponding to the second query result. Based on the similarity score between the text and the image corresponding to the second query result, the enhanced text feature data is used to perform semantic enhancement on the enhanced image feature data to obtain text-enhanced image feature data; Based on the similarity score between the image and the text corresponding to the first query result and the similarity score between the text and the image corresponding to the second query result, the interaction gating value between the image and the text is calculated. Using the interactive gating value, the text-enhanced image feature data is weighted, and the weighted text-enhanced image feature data and the image-enhanced text feature data are combined in the form of residuals to complete bidirectional interactive fusion and obtain image-text fusion feature data.

[0011] In one possible design, the image-text fusion feature data is used as the output for updating the parameters of a large model, and the large model is trained to update its parameters, including: The text data and the image data are masked respectively to obtain parameter update training samples; The parameter update training samples are used as the input for updating the parameters of the large model, and the image-text fusion feature data are used as the output for updating the parameters. The parameters in the large model are then trained for parameter updates.

[0012] In a second aspect, the present invention provides a multimodal data fusion system for training large models, applied to the multimodal data fusion method for training large models as described in the first aspect or any possible design of the first aspect, comprising: The feature extraction unit is used to acquire text data and image data, extract features from the text data using a pre-trained semantic feature extraction model to obtain text feature data, and extract features from the image data using the semantic feature extraction model to obtain image feature data, wherein the image feature data includes local image feature data and global image feature data. The feature enhancement unit is used to generate supplementary text feature data based on the global image feature data using a cooperative autovariable encoder, perform weighted fusion of the text feature data and the supplementary text feature data to obtain enhanced text feature data, and perform weighted fusion of the local image feature data and the global image feature data to obtain enhanced image feature data. The feature fusion unit is used to perform cross-querying and semantic enhancement on the enhanced text feature data and the enhanced image feature data using a cross-attention mechanism to complete bidirectional interactive fusion and obtain image-text fusion feature data. The large model training unit is used to use the image-text fusion feature data as the output of the large model parameter update to train the large model.

[0013] Thirdly, the present invention provides an electronic device comprising a memory, a processor, and a transceiver connected in sequence and communication, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the multimodal data fusion method for training large models as described in the first aspect or any possible design of the first aspect.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, perform the multimodal data fusion method for training large models as described in the first aspect or any possible design of the first aspect.

[0015] Fifthly, the present invention provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform a multimodal data fusion method for training large models as described in the first aspect or any possible design of the first aspect.

[0016] Beneficial Effects: This invention provides a multimodal data fusion method for training large models, comprising: First, acquiring text data and image data; extracting features from the text data using a pre-trained semantic feature extraction model to obtain text feature data; and extracting features from the image data using the same semantic feature extraction model to obtain image feature data, wherein the image feature data includes local image feature data and global image feature data; Second, based on the global image feature data, generating supplementary text feature data using a collaborative autovariable encoder; weightedly fusing the text feature data and the supplementary text feature data to obtain enhanced text feature data; and weightedly fusing the local image feature data and the global image feature data to obtain enhanced image feature data; Then, using a cross-attention mechanism, performing cross-querying and semantic enhancement on the enhanced text feature data and the enhanced image feature data to complete bidirectional interactive fusion, obtaining image-text fused feature data; Finally, using the image-text fused feature data as the parameter update output of a large model to train the large model for parameter updates. A pre-trained semantic feature extraction model is used to extract detailed features from text and image data, resulting in text feature data and image feature data. A collaborative autovariable encoder is used to generate supplementary text feature data from the image data. The supplementary text feature data is then used to perform weighted fusion of the text feature data. Local and global image feature data are then fused to form pre-fused enhanced features, ensuring that subsequent fused data can fully utilize sufficient detailed features. Through interactive queries and semantic enhancement, image and text features are fused. During the training of the large model, rich fused semantic information is used to update the parameters of the large model to improve its image and text recognition capabilities. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the multimodal data fusion method for training large models provided in an embodiment of the present invention. Figure 2 This is a functional structure diagram of a multimodal data fusion system for large model training provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0019] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0020] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0021] Example: like Figure 1 As shown, the first aspect of this embodiment provides a multimodal data fusion method for training large models, which may include, but is not limited to, the following steps: S1. Acquire text data and image data, extract features from the text data using a pre-trained semantic feature extraction model to obtain text feature data, and extract features from the image data using the semantic feature extraction model to obtain image feature data, wherein the image feature data includes local image feature data and global image feature data; In one possible implementation, the pre-training method of the semantic feature extraction model in step S1 may include, but is not limited to, the following steps S101-S108: S101. Obtain raw text data from a massive dataset as raw pre-training text samples, and perform masking processing on the raw pre-training text samples according to a first preset ratio to obtain pre-training text samples. S102. Select the BERT model as the base model, take the pre-trained text samples as input, take the original text data as output, and pre-train the BERT model based on self-supervised learning to obtain a text semantic feature extraction sub-model. S103. Obtain raw image data from a massive dataset and obtain the real labels corresponding to the raw image data, wherein the real labels of the raw image data are used to represent the real image category information and complete semantic information corresponding to the raw image data; S104. Use the original image data as the first pre-trained image sample, and use the real label of the original image data as the first pre-trained image semantic feature; S105. Select the ResNet model as the base model, take the first pre-trained image sample as the input, take the semantic features of the first pre-trained image as the output, and form a one-to-one corresponding input and output pair. Based on supervised learning, pre-train the ResNet model to obtain the first image semantic feature extraction sub-model. S106. Perform image block processing on the original image data to obtain multiple image blocks, use each image block as the original second pre-training image sample, and perform masking processing on the original second pre-training image sample according to the second preset ratio to obtain the second pre-training image sample. S107. Select the ViT model as the base model, take the second pre-trained image sample as the input, take the real label of the original image data as the output, and pre-train the ViT model based on self-supervised learning to obtain the second image semantic feature extraction sub-model. S108. The text semantic feature extraction sub-model is used as the text feature extraction encoder, the first image semantic feature extraction sub-model is used as the local image feature extraction encoder, and the second image semantic feature extraction sub-model is used as the global image feature extraction encoder. Based on the text feature extraction encoder, the local image feature extraction encoder, and the global image feature extraction encoder, a semantic feature extraction model is formed.

[0022] It should be noted that BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model used to learn language rules on large-scale texts through self-supervised pre-training, and then adapt to specific tasks through supervised fine-tuning. This pre-training mode avoids the expensive cost of training from scratch, making it a highly suitable method in the field of natural language processing. By introducing the self-attention mechanism of the Transformer encoder, the BERT model can achieve omnidirectional contextual joint modeling of pre-trained text samples. ResNet (Residual Network), proposed by Microsoft Research, addresses the problems of gradient vanishing, exploding, and performance degradation in deep neural network training. It provides a deeper understanding of feature extraction and introduces residual connections, allowing gradients to propagate directly across layers, enabling the network to "focus on learning residuals," thus improving the depth of feature extraction in neural networks. Therefore, it is very suitable for the local image feature extraction processing in this application, obtaining richer semantic feature information. The ViT (Vision Transformer) model is a vision model based on the Transformer architecture. It directly derives the global relationships between image patches by introducing a self-attention mechanism, making it particularly suitable for extracting global and complete semantic features from large-scale image data in this embodiment.

[0023] In one possible implementation, the semantic feature extraction model is a multi-head model, including a text input head and an image input head, and also including a text feature output head, a local image feature output head, and a global image feature output head; Accordingly, in step S1, acquiring text data and image data, extracting features from the text data using a pre-trained semantic feature extraction model to obtain text feature data, and extracting features from the image data using the same semantic feature extraction model to obtain image feature data, can be decomposed into, but is not limited to, the following steps S11-S13, specifically including: S11. Extract interrelated text and image data from a massive dataset; S12. Perform data cleaning and standardization on the text data, input the processed text data into the semantic feature extraction model through the text input head, extract features from the text data using the text feature extraction encoder, and output text feature data through the text feature output head, wherein the text feature data includes word vectors and sentence vectors; S13. The image data is subjected to size unification and normalization processing. The processed image data is input into the semantic feature extraction model through the image input head. The local image feature extraction encoder and the global image feature extraction encoder are used to extract features from the image data. Local image feature data is output through the local image feature output head, and global image feature data is output through the global image feature output head. The local image feature data includes feature maps, and the global image feature data includes image semantic vectors.

[0024] S2. Based on the global image feature data, supplementary text feature data is generated using a collaborative autovariable encoder. The text feature data and the supplementary text feature data are weighted and fused to obtain enhanced text feature data. The local image feature data and the global image feature data are weighted and fused to obtain enhanced image feature data. In one possible implementation, step S2, based on the global image feature data, generates supplementary text feature data using a cooperative autovariable encoder. This can be decomposed into, but is not limited to, the following steps S21-S23, specifically including: S21. The global image feature data is used as high-dimensional global image features. The high-dimensional global image features are then reduced in dimensionality using the convolutional layer and average pooling layer of the cooperative autovariable encoder to obtain the low-dimensional global image features after dimensionality reduction and compression. S22. Using the low-dimensional coding network of the cooperative autovariable encoder, the low-dimensional global image features are encoded to obtain a Gaussian distribution of low-dimensional image features, wherein the Gaussian distribution of low-dimensional image features is defined by the mean and logarithmic variance of the low-dimensional image features. S23. Input the text feature data into the convolutional layer of the collaborative autovariable encoder to obtain high-dimensional text features; S24. Perform feature concatenation on the high-dimensional global image features and the high-dimensional text features to obtain high-dimensional image-text fusion features; S25. Input the Gaussian distribution of the low-dimensional image features into the joint high-dimensional coding network of the collaborative autovariable encoder to perform joint coding processing on the Gaussian distribution of the low-dimensional image features and the high-dimensional image-text fusion features to obtain the Gaussian distribution of the high-dimensional image-text fusion features, wherein the Gaussian distribution of the high-dimensional image-text fusion features is defined by the mean and logarithmic variance of the high-dimensional image-text fusion features; S26. Using reparameterized sampling, latent variables are sampled from the Gaussian distribution of the low-dimensional image features and the Gaussian distribution of the high-dimensional image-text fusion features, respectively, to obtain latent variables of the low-dimensional image features and latent variables of the high-dimensional image-text fusion features. S27. Based on the preset high-dimensional variable weighting coefficients in the collaborative autovariable encoder, the low-dimensional image feature latent variable and the high-dimensional image-text fusion feature latent variable are weighted and summed to obtain the joint latent variable of high- and low-dimensional fusion features. S28. Decode the high- and low-dimensional fusion features and the joint latent variables to generate high- and low-dimensional fusion features, and use the high- and low-dimensional fusion features as supplementary text feature data.

[0025] It should be noted that reparameterization trick is a key technique applied in deep learning, primarily used to address the gradient non-differentiability problem in model training involving stochastic operations. It decomposes the generation process of random variables into deterministic and random parts, enabling backpropagation to proceed normally. Therefore, when used for latent variable sampling in this embodiment, the stability of the latent variable sampling is crucial.

[0026] In one possible implementation, step S2 involves weighted fusion of the text feature data and the supplementary text feature data to obtain enhanced text feature data. This can be, but is not limited to, decomposed into the following steps S29-S212, specifically including: S29. Obtain a preset single-modal fusion model, wherein the single-modal fusion model includes a text feature processing convolutional block and an image feature processing convolutional block, wherein the text feature processing convolutional block is used to process text feature data and supplement text feature data, and the image feature processing convolutional block is used to process local image feature data and global image feature data; S210. Using the text feature processing convolutional block in the single-modal fusion model, semantic features are extracted from the text feature data and the supplementary text feature data to obtain text semantic feature data and supplementary text semantic feature data; S211. Obtain the preset text feature weight matrix in the single-modal fusion model, wherein the text feature weight matrix includes the preset weights assigned to the text semantic feature data and the supplementary text semantic feature data in the text feature processing convolution block; S212. Based on the text feature weight matrix, the text semantic feature data and the supplementary text semantic feature data are weighted and summed to obtain enhanced text feature data; Accordingly, in one possible implementation, step S2, which involves weighted fusion of the local image feature data and the global image feature data to obtain enhanced image feature data, can be decomposed into, but is not limited to, the following steps S213-S215, specifically including: S213. Using the image feature processing convolutional block in the single-modal fusion model, semantic features are extracted from the local image feature data and the global image feature data to obtain local image semantic feature data and global image semantic feature data; S214. Obtain the preset image feature weight matrix in the single-modal fusion model, wherein the image feature weight matrix includes the preset weights assigned to the local image semantic feature data and the global image semantic feature data in the image feature processing convolution block; S215. Based on the image feature weight matrix, the local image semantic feature data and the global image semantic feature data are weighted and summed to obtain enhanced image feature data.

[0027] S3. Using a cross-attention mechanism, cross-query and semantic enhancement are performed on the enhanced text feature data and the enhanced image feature data to complete bidirectional interactive fusion and obtain image-text fusion feature data; In one possible implementation, step S3 utilizes a cross-attention mechanism to perform cross-querying and semantic enhancement on the enhanced text feature data and the enhanced image feature data to complete bidirectional interactive fusion and obtain image-text fusion feature data. This can be decomposed into, but is not limited to, the following steps S31-S36, specifically including: S31. Using the enhanced text feature data as a query vector, query the enhanced image feature data to obtain the first query result of the enhanced text feature data on the enhanced image feature data, and calculate the image-to-text similarity score corresponding to the first query result. S32. Based on the image-to-text similarity score corresponding to the first query result, semantic enhancement is performed on the enhanced text feature data using the enhanced image feature data to obtain image-enhanced text feature data; S33. Using the enhanced image feature data as a query vector, query the enhanced text feature data to obtain a second query result of the enhanced image feature data on the enhanced text feature data, and calculate the similarity score between the text and the image corresponding to the second query result; S34. Based on the similarity score between the text and the image corresponding to the second query result, semantic enhancement is performed on the enhanced image feature data using the enhanced text feature data to obtain text-enhanced image feature data; S35. Calculate the interaction gating value between the image and the text based on the similarity score between the image and the text corresponding to the first query result and the similarity score between the text and the image corresponding to the second query result; S36. Using the interactive gating value, the text-enhanced image feature data is weighted, and the weighted text-enhanced image feature data and the image-enhanced text feature data are combined in the form of residuals to complete bidirectional interactive fusion and obtain image-text fusion feature data.

[0028] S4. Use the image-text fusion feature data as the output of the large model parameter update to train the large model.

[0029] In one possible implementation, step S4, where the image-text fusion feature data is used as the output for updating the parameters of the large model, and the large model is trained to update its parameters, can be decomposed into, but is not limited to, the following steps S41-S42, specifically including: S41. Perform masking processing on the text data and the image data respectively to obtain parameter update training samples; S42. Use the parameter update training samples as the input for the large model parameter update, and use the image-text fusion feature data as the output for the parameter update, to perform parameter update training on the parameters in the large model.

[0030] like Figure 2 As shown, the second aspect of this embodiment provides a hardware system for implementing the multimodal data fusion method for large model training described in the first aspect of the embodiment, including: The feature extraction unit is used to acquire text data and image data, extract features from the text data using a pre-trained semantic feature extraction model to obtain text feature data, and extract features from the image data using the semantic feature extraction model to obtain image feature data, wherein the image feature data includes local image feature data and global image feature data. The feature enhancement unit is used to generate supplementary text feature data based on the global image feature data using a cooperative autovariable encoder, perform weighted fusion of the text feature data and the supplementary text feature data to obtain enhanced text feature data, and perform weighted fusion of the local image feature data and the global image feature data to obtain enhanced image feature data. The feature fusion unit is used to perform cross-querying and semantic enhancement on the enhanced text feature data and the enhanced image feature data using a cross-attention mechanism to complete bidirectional interactive fusion and obtain image-text fusion feature data. The large model training unit is used to use the image-text fusion feature data as the output of the large model parameter update to train the large model.

[0031] The working process, working details and technical effects of the system provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0032] like Figure 3 As shown, the third aspect of this embodiment provides an electronic device, including: a memory, a processor, and a transceiver that are sequentially and communicatively connected, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the multimodal data fusion method for large model training as described in the first aspect of the embodiment.

[0033] For specific examples, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state.

[0034] In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. For example, the processor may not be limited to microprocessors of the STM32F105 series, reduced instruction set computer (RISC) microprocessors, x86 architecture processors, or processors with integrated neural network processing units (NPUs). The transceiver may be, but is not limited to, a Wi-Fi transceiver, a Bluetooth transceiver, a General Packet Radio Service (GPRS) transceiver, a ZigBee (a low-power LAN protocol based on the IEEE 802.15.4 standard) transceiver, a 3G transceiver, a 4G transceiver, and / or a 5G transceiver. Furthermore, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0035] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0036] The fourth aspect of this embodiment provides a storage medium that stores instructions for a multimodal data fusion method for training a large model as described in the first aspect of the embodiment. That is, the storage medium stores instructions that, when executed on a computer, perform the multimodal data fusion method for training a large model as described in the first aspect of the embodiment.

[0037] The storage medium refers to a carrier for storing data, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0038] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0039] The fifth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the multimodal data fusion method for training large models as described in the first aspect of this embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0040] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal data fusion method for training large models, characterized in that, include: Text data and image data are acquired. The text data is then processed by a pre-trained semantic feature extraction model to obtain text feature data. The image data is then processed by the same semantic feature extraction model to obtain image feature data. The image feature data includes local image feature data and global image feature data. Based on the global image feature data, supplementary text feature data is generated using a collaborative autovariable encoder. The text feature data and the supplementary text feature data are then weighted and fused to obtain enhanced text feature data. The local image feature data and the global image feature data are then weighted and fused to obtain enhanced image feature data. By utilizing the cross-attention mechanism, cross-querying and semantic enhancement are performed on the enhanced text feature data and the enhanced image feature data to complete bidirectional interactive fusion and obtain image-text fusion feature data; The image-text fusion feature data is used as the output of the large model parameter update to train the large model.

2. The multimodal data fusion method for large model training according to claim 1, characterized in that, The pre-training method for the semantic feature extraction model includes: The original text data is obtained from the ultra-large-scale dataset as the original pre-training text sample. The original pre-training text sample is then masked according to a first preset ratio to obtain the pre-training text sample. The BERT model is selected as the base model. The pre-trained text samples are used as input and the original text data is used as output. The BERT model is pre-trained based on self-supervised learning to obtain a text semantic feature extraction sub-model. Obtain raw image data from a massive dataset and obtain the real labels corresponding to the raw image data. The real labels of the raw image data are used to represent the real image category information and complete semantic information corresponding to the raw image data. The original image data is used as the first pre-trained image sample, and the true label of the original image data is used as the first pre-trained image semantic feature. The ResNet model is selected as the base model. The first pre-trained image sample is used as the input and the semantic features of the first pre-trained image are used as the output, forming a one-to-one input and output pair. Based on supervised learning, the ResNet model is pre-trained to obtain the first image semantic feature extraction sub-model. The original image data is processed into image blocks to obtain multiple image blocks. Each image block is used as an original second pre-training image sample. The original second pre-training image sample is masked according to a second preset ratio to obtain a second pre-training image sample. The ViT model is selected as the base model. The second pre-trained image sample is used as the input and the real label of the original image data is used as the output. The ViT model is pre-trained based on self-supervised learning to obtain the second image semantic feature extraction sub-model. The text semantic feature extraction sub-model is used as the text feature extraction encoder, the first image semantic feature extraction sub-model is used as the local image feature extraction encoder, and the second image semantic feature extraction sub-model is used as the global image feature extraction encoder. Based on the text feature extraction encoder, the local image feature extraction encoder, and the global image feature extraction encoder, a semantic feature extraction model is formed.

3. The multimodal data fusion method for large model training according to claim 2, characterized in that, The semantic feature extraction model is a multi-head model, including a text input head and an image input head, as well as a text feature output head, a local image feature output head, and a global image feature output head; Accordingly, acquiring text data and image data, extracting features from the text data using a pre-trained semantic feature extraction model to obtain text feature data, and extracting features from the image data using the same semantic feature extraction model to obtain image feature data, includes: Extracting interconnected text and image data from massive datasets; The text data is cleaned and standardized. The processed text data is then input into the semantic feature extraction model through the text input head. The text feature extraction encoder is used to extract features from the text data. The text feature output head outputs text feature data, which includes word vectors and sentence vectors. The image data is subjected to size unification and normalization processing. The processed image data is then input into the semantic feature extraction model through the image input head. The local image feature extraction encoder and the global image feature extraction encoder are used to extract features from the image data. Local image feature data is output through the local image feature output head, and global image feature data is output through the global image feature output head. The local image feature data includes feature maps, and the global image feature data includes image semantic vectors.

4. The multimodal data fusion method for large model training according to claim 1, characterized in that, Based on the global image feature data, supplementary text feature data is generated using a cooperative autovariable encoder, including: The global image feature data is used as high-dimensional global image features. The high-dimensional global image features are then reduced in dimensionality using the convolutional layer and average pooling layer of the cooperative autovariable encoder to obtain the dimensionality-reduced and compressed low-dimensional global image features. The low-dimensional global image features are encoded using the low-dimensional coding network of the cooperative autovariable encoder to obtain a Gaussian distribution of low-dimensional image features, wherein the Gaussian distribution of low-dimensional image features is defined by the mean and logarithmic variance of the low-dimensional image features. The text feature data is input into the convolutional layer of the collaborative autovariable encoder to obtain high-dimensional text features; The high-dimensional global image features and the high-dimensional text features are concatenated to obtain high-dimensional image-text fusion features; The low-dimensional image feature Gaussian distribution is input into the joint high-dimensional coding network of the collaborative autovariable encoder to jointly encode the low-dimensional image feature Gaussian distribution and the high-dimensional image-text fusion feature to obtain the high-dimensional image-text fusion feature Gaussian distribution, wherein the high-dimensional image-text fusion feature Gaussian distribution is defined by the mean and logarithmic variance of the high-dimensional image-text fusion feature. Using reparameterized sampling, latent variables are sampled from the Gaussian distribution of the low-dimensional image features and the Gaussian distribution of the high-dimensional image-text fusion features, respectively, to obtain the latent variables of the low-dimensional image features and the latent variables of the high-dimensional image-text fusion features. Based on the preset high-dimensional variable weighting coefficients in the collaborative autovariable encoder, the low-dimensional image feature latent variable and the high-dimensional image-text fusion feature latent variable are weighted and summed to obtain the joint latent variable of high- and low-dimensional fusion features. The high- and low-dimensional fused features are decoded together with the latent variables to generate high- and low-dimensional fused features, which are then used as supplementary text feature data.

5. The multimodal data fusion method for large model training according to claim 1, characterized in that, The text feature data and the supplementary text feature data are weighted and fused to obtain enhanced text feature data, including: Obtain a preset single-modal fusion model, wherein the single-modal fusion model includes a text feature processing convolutional block and an image feature processing convolutional block, wherein the text feature processing convolutional block is used to process text feature data and supplement text feature data, and the image feature processing convolutional block is used to process local image feature data and global image feature data; Using the text feature processing convolutional block in the unimodal fusion model, semantic features are extracted from the text feature data and the supplementary text feature data to obtain text semantic feature data and supplementary text semantic feature data; Obtain the preset text feature weight matrix in the single-modal fusion model, wherein the text feature weight matrix includes the preset weights assigned to the text semantic feature data and the supplementary text semantic feature data in the text feature processing convolution block; Based on the text feature weight matrix, the text semantic feature data and the supplementary text semantic feature data are weighted and summed to obtain the enhanced text feature data. Accordingly, weighted fusion of the local image feature data and the global image feature data to obtain enhanced image feature data includes: Using the image feature processing convolutional block in the unimodal fusion model, semantic features are extracted from the local image feature data and the global image feature data to obtain local image semantic feature data and global image semantic feature data; Obtain the preset image feature weight matrix in the single-modal fusion model, wherein the image feature weight matrix includes the preset weights assigned to the local image semantic feature data and the global image semantic feature data in the image feature processing convolution block; Based on the image feature weight matrix, the local image semantic feature data and the global image semantic feature data are weighted and summed to obtain enhanced image feature data.

6. The multimodal data fusion method for large model training according to claim 1, characterized in that, Using a cross-attention mechanism, the enhanced text feature data and the enhanced image feature data are cross-queried and semantically enhanced to achieve bidirectional interactive fusion, resulting in image-text fusion feature data, including: The enhanced text feature data is used as a query vector to query the enhanced image feature data to obtain the first query result of the enhanced text feature data on the enhanced image feature data, and the similarity score between the image and the text corresponding to the first query result is calculated. Based on the image-to-text similarity score corresponding to the first query result, the enhanced image feature data is used to semantically enhance the enhanced text feature data to obtain image-enhanced text feature data. The enhanced image feature data is used as a query vector to query the enhanced text feature data, so as to obtain the second query result of the enhanced image feature data on the enhanced text feature data, and calculate the similarity score between the text and the image corresponding to the second query result. Based on the similarity score between the text and the image corresponding to the second query result, the enhanced text feature data is used to perform semantic enhancement on the enhanced image feature data to obtain text-enhanced image feature data; Based on the similarity score between the image and the text corresponding to the first query result and the similarity score between the text and the image corresponding to the second query result, the interaction gating value between the image and the text is calculated. Using the interactive gating value, the text-enhanced image feature data is weighted, and the weighted text-enhanced image feature data and the image-enhanced text feature data are combined in the form of residuals to complete bidirectional interactive fusion and obtain image-text fusion feature data.

7. The multimodal data fusion method for large model training according to claim 1, characterized in that, The image-text fusion feature data is used as the output for updating the parameters of the large model. The large model is then trained with updated parameters, including: The text data and the image data are masked respectively to obtain parameter update training samples; The parameter update training samples are used as the input for updating the parameters of the large model, and the image-text fusion feature data are used as the output for updating the parameters. The parameters in the large model are then trained for parameter updates.

8. A multimodal data fusion system for training large models, applied to the multimodal data fusion method for training large models as described in any one of claims 1 to 7, characterized in that, include: The feature extraction unit is used to acquire text data and image data, extract features from the text data using a pre-trained semantic feature extraction model to obtain text feature data, and extract features from the image data using the semantic feature extraction model to obtain image feature data, wherein the image feature data includes local image feature data and global image feature data. The feature enhancement unit is used to generate supplementary text feature data based on the global image feature data using a cooperative autovariable encoder, perform weighted fusion of the text feature data and the supplementary text feature data to obtain enhanced text feature data, and perform weighted fusion of the local image feature data and the global image feature data to obtain enhanced image feature data. The feature fusion unit is used to perform cross-querying and semantic enhancement on the enhanced text feature data and the enhanced image feature data using a cross-attention mechanism to complete bidirectional interactive fusion and obtain image-text fusion feature data. The large model training unit is used to use the image-text fusion feature data as the output of the large model parameter update to train the large model.

9. An electronic device, characterized in that, The device includes a memory, a processor, and a transceiver that are sequentially and communicatively connected. The memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the multimodal data fusion method for training large models as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or the instructions are executed by the computer, they implement the multimodal data fusion method for training large models as described in any one of claims 1 to 7.

Images