Multi-modal data fusion modeling method and system based on multi-task learning

Abstract

The invention relates to the technical field of artificial intelligence, and provides a multi-modal data fusion modeling method based on multi-task learning, and the method comprises the steps: respectively extracting image and text features in data, and carrying out the fusion to obtain first fusion data; respectively inputting the fused data into a Transform decoder and at least two weight layers for processing, and fusing the fused data with the original fused data again through a self-adaptive gating layer to obtain second fused data; and finally, based on the second fusion data, respectively calculating classification loss and comparison loss, and when the two types of loss meet preset conditions, outputting a final multi-modal data fusion model. The invention further discloses a system. According to the method and the system, the semantic alignment between the image and the text is enhanced, the cross-modal semantic problem is effectively solved, and the understanding ability of the model to the complex semantic relationship is improved. The model is enhanced to have adaptability in different task scenes, the generalization ability and robustness of the model are remarkably improved, and data noise is reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-modal data fusion modeling method and system based on multi-task learning. Background Art

[0002] With the booming development of artificial intelligence and the widespread use of large models, data diversity and complexity have reached unprecedented levels. Humans perceive the world through multiple senses, including vision, hearing, and touch. Correspondingly, in the field of artificial intelligence, data is presented in multiple modalities, including images, text, audio, and video. Data from different modalities is like a treasure trove of rich resources, each containing a wealth of complementary information, providing vast opportunities for the development of artificial intelligence. Consequently, cross-modal learning and fusion tasks have emerged and become a research focus in numerous fields. For example, in the field of medical diagnosis, medical data is abundant and diverse, including medical images such as X-rays, CT scans, and MRIs, as well as textual information such as medical records and doctor's diagnostic notes. Medical images provide a visual representation of the internal structure of the human body and the condition of its pathology, while medical records provide detailed information about a patient's symptoms, medical history, and treatment.

[0003] CN120146549A discloses a construction safety risk warning method and system based on cross-modal visual language retrieval, comprising the following steps: (1) understanding each element of a given triple and extracting aligned image and text features respectively, providing an alignment basis for subsequent cross-modal combination, and based on the image-text alignment feature, decoupling the data distribution, information expression mode and semantics of the image and text, and providing multi-granularity decoupled feature information for subsequent cross-modal combination; (2) completing the combined feature extraction of multi-modal query from the multi-granularity level for the extracted multi-granularity decoupled feature information, and obtaining combined semantic information at different granularity details; (3) performing alignment between the multi-granularity combined feature and the multi-granularity target feature, pushing the combined feature to approach the target image feature from the multi-granularity level, realizing cross-modal alignment at the multi-granularity detail level, and aligning the semantic information between the multi-modal combination and the target image; (4) after aligning the semantic information between the multi-modal combination and the target image, obtaining a text-image multi-modal combination retrieval model, using the model to process the multi-modal query information, retrieving the corresponding risk target image, matching the risk content and issuing a warning.

[0004] CN119311911B discloses a cross-modal image-text retrieval method based on deep learning. The method includes the following steps: Step 1: Feature extraction stage, extracting deep features from images and texts to ensure that cross-modal data is semantically aligned in a unified feature space; Step 2: Relevance scoring stage, evaluating the relevance between images and texts based on a fine-grained attention mechanism; Step 3: Optimization stage, optimizing retrieval performance through specific loss functions and model strategies. The feature extraction stage aims to extract deep feature representations from images and texts so as to achieve effective cross-modal semantic alignment in subsequent processing. The text is encoded through the BERT model to obtain its rich semantic features. At the same time, Faster R-CNN and ResNet-101 are used to analyze the image and extract the visual features of the image to ensure that the features of the two modalities can be compared and aligned in a unified feature space; The relevance scoring stage aims to evaluate and quantify the similarity between image features and text features, generate a relevance scoring matrix, and use I2T Attention and T2IAttention mechanisms, the model can identify fine-grained associations between images and texts, capture the most relevant parts of the image to the text and the most relevant words in the text to the image, and thus calculate a similarity score for each image-text pair, providing a basis for the final retrieval results; the optimization stage aims to fine-tune the model through similarity scoring and ranking loss to optimize the performance of cross-modal retrieval. The model parameters are adjusted according to the data of the query image or text to ensure that the model can capture the unique characteristics of the cross-modal data distribution and maximize the similarity of related image and text pairs while minimizing the similarity of unrelated pairs; the optimization stage includes the definition and calculation of the loss function, the calculation of the loss and back propagation; the definition and calculation of the loss function are based on Triplet The Loss principle guides model learning by calculating the distance difference between positive samples and negative samples; the loss calculation and backpropagation are performed by using the compute_loss method to calculate the loss L, and the calculation formula of L is as follows: wherein represents the diagonal element of the i-th row, and margin is a predefined boundary value used to control the distance between positive and negative samples; first extract the elements on the diagonal, then find the maximum value of each row and column, and calculate the loss, and finally backpropagate the loss function to update the model parameters; the optimization stage also includes a fast and slow model strategy, the fast model is used to quickly screen out potential related images or texts, and the slow model is used to fine-tune and optimize the model parameters to improve the accuracy and relevance of the retrieval.

[0005] Existing technologies have difficulty building efficient joint modeling frameworks to achieve collaborative representation of image and text data. Cross-modal semantic alignment remains a challenge, and deep semantic relationship mining is insufficient. Traditional models cannot fully exploit the complementary information between modalities, and existing fusion strategies lack flexibility and adaptability. Summary of the Invention

[0006] Long-term practice has shown that traditional unimodal models struggle to fully exploit the complementary information between modalities, resulting in insufficient generalization and an inability to effectively address the growing demand for multimodal data processing in real-world applications. Multimodal learning methods often fail to adequately model the deep semantic relationships between images and text, and fusion strategies lack flexibility and adaptability. In complex scenarios like medical diagnosis and remote sensing analysis, models suffer from poor robustness and inability to generalize sufficiently to meet practical application requirements.

[0007] In view of this, the present invention aims to propose a multimodal data fusion modeling method based on multi-task learning, comprising:

[0008] Step S1, establishing a feature extraction unit, wherein the feature extraction unit includes a text encoder module and an image encoder module, wherein the image encoder module includes a DenseNet module for extracting image features from data; and the text encoder module includes a BERT module for extracting text features from data;

[0009] Step S2, fusing the image features extracted by the image encoder module and the text features extracted by the text encoder module to obtain first fused data;

[0010] Step S3: input the first fused data into the Transformer decoder and the weight layer respectively; the first fused data is used as the input of the Transformer decoder, and is passed through multiple self-attention layers and cross-attention layers in the Transformer decoder to obtain first processed data; the first fused data is input into at least two layers of the weight layers to obtain second processed data, and the second processed data is further fused with the first fused data to obtain third processed data; the first processed data, the third processed data, and the first fused data are input into the adaptive gating layer to obtain second fused data;

[0011] Step S4: input the fusion features in the second fusion data into the predicted label vector, and calculate the classification loss and average loss; calculate the contrast loss and average loss of the aligned text features and aligned image features in the second fusion data; if the classification loss, average loss and contrast loss all meet the preset conditions, output a multimodal data fusion model based on multi-task learning.

[0012] Preferably, the first processed data, the third processed data, and the first fused data are input into the adaptive gating layer for weighted fusion. If the input is X:

[0013] X∈R {B×C×D}

[0014] Where R is the real number domain, B is the number of samples processed in parallel during each round of training or inference, C is the number of channels, corresponding to the number of feature channels of each modality before fusion, and D is the length of the feature dimension corresponding to each channel;

[0015] Input X first passes through the Transformer encoder to obtain the global context to obtain H;

[0016] H = TransformerEncoder(X)

[0017] Perform local enhancement through ResNet to obtain the output P of the ResNet module;

[0018] P=X+f(X)

[0019] Among them, f contains two layers of one-dimensional convolution, batch normalization and ReLU activation;

[0020] The H and P inputs are concatenated and input into the adaptive gating layer to calculate the adjustment weight r:

[0021] F=concat(P,H)

[0022] A=MLP(AdaptiveAvgPool1D(F))

[0023] r = Softmax(A)

[0024] The fused H′ is,

[0025] H′=P+r·H

[0026] Among them, r is used to control the residual enhancement of the Transformer feature to the original residual information.

[0027] Preferably, the fused s is,

[0028] S=r·P+(1-r)·H′

[0029] Among them, r is used to control the dominant mode of the final output fusion and adjust the rebalancing of semantics and structure.

[0030] Preferably, in step S4, the classification loss is calculated as,

[0031]

[0032] Among them, N is the total number of samples, G is the number of label types, and y n，g is the true label value of the nth sample in class g, O n is the output vector of the nth sample;

[0033] The output matrix of image features is,

[0034] P∈R N×D

[0035] Among them, the image feature output matrix is ​​I∈R N×D , the output matrix of text features is T∈R N×D , normalize the two to get the similarity matrix S of image features and text features TI for,

[0036]

[0037] Among them, τ is the temperature parameter, which is used to adjust the smoothness of the distribution.

[0038] Preferably, the image features extracted by the image encoder module are subjected to dimensionality reduction and transformation on the extracted high-dimensional features through a fully connected layer, so that the output image features are consistent with the input dimensions of subsequent modules;

[0039] The input text is converted into a token sequence by a word segmenter, and after adding position code and type tag, it is input into the text encoder module for text feature extraction.

[0040] Preferably, in step S3, the image feature is I∈R {B×C×D} , the text feature is R∈R {B×D} , the first fused data are respectively input into the Transformer decoder, and after fusion, the result is Output = TransformerDecoder (I, T).

[0041] The present invention also provides a system for the multi-modal data fusion modeling method based on multi-task learning as described above, the system comprising:

[0042] A feature extraction unit, the feature extraction unit comprising a text encoder module and an image encoder module, the image encoder module comprising a DenseNet module for extracting image features from the data; the text encoder module comprising a BERT module for extracting text features from the data;

[0043] a first fusion unit, configured to fuse the image features extracted by the image encoder module and the text features extracted by the text encoder module to obtain first fused data;

[0044] The second fusion unit is used to input the first fused data into the Transformer decoder and the weight layer respectively; the first fused data is used as the input of the Transformer decoder, and is passed through multiple self-attention layers and cross-attention layers in the Transformer decoder to obtain first processed data; the first fused data is respectively input into at least two layers of the weight layers to obtain second processed data, and the second processed data is further fused with the first fused data to obtain third processed data; the first processed data, the third processed data, and the first fused data are input into the adaptive gating layer to obtain second fused data;

[0045] The multi-task learning optimization unit is used to input the fusion features in the second fusion data into the predicted label vector and calculate the classification loss and average loss; calculate the contrast loss and average loss of the aligned text features and aligned image features in the second fusion data; if the classification loss, average loss and contrast loss all meet preset conditions, output a multimodal data fusion model based on multi-task learning.

[0046] Preferably, the second fusion unit further includes a fusion module based on a gating mechanism, and the fusion module based on the gating mechanism includes a Transformer encoder and a ResNet model, wherein the Transformer encoder is used for global context information modeling and the ResNet model is used for enhancing local feature learning.

[0047] The present invention provides an electronic device, at least one processor; and

[0048] a memory communicatively connected to the at least one processor; wherein,

[0049] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0050] The present invention provides a machine-readable storage medium having instructions stored thereon, wherein the instructions are used to enable a machine to execute the method of the present application as described above.

[0051] The multimodal data fusion modeling method based on multi-task learning disclosed in the present invention first uses the DenseNet module and the BERT module to extract the image and text features in the data respectively, and obtains the first fused data after fusion; then inputs them into the Transformer decoder and at least two weight layers for processing, and fuses them again with the original fused data through the adaptive gating layer to obtain the second fused data; finally, based on the second fused data, the classification loss and the contrast loss are calculated respectively, and when both types of losses meet the preset conditions, the final multimodal data fusion model is output. The present invention also discloses a system for the multimodal data fusion modeling method based on multi-task learning as described above. The method and system first perform preliminary feature fusion, and then combine the processing results of the Transformer decoder and the weight layer, dynamically adjust the weight of each part of the information through the adaptive gating layer, strengthen the semantic alignment between the image and the text, effectively solve the cross-modal semantic problem, and enhance the model's ability to understand complex semantic relationships. Through optimizations such as classification loss, contrast loss, and average loss, the model can learn simultaneously in classification tasks and cross-modal alignment tasks, enhancing the model's adaptability to different task scenarios, significantly improving the model's generalization ability and robustness in complex scenarios such as medical diagnosis and remote sensing analysis, and reducing data noise.

[0052] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0054] In the attached figure:

[0055] Figure 1 Schematic diagram of a multimodal data fusion modeling method based on multi-task learning according to an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the structure of a fusion module based on a gating mechanism in a system of a multimodal data fusion modeling method based on multi-task learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0058] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0059] It should be noted that the terms "first," "second," "third," etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate to facilitate the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or are inherent to these processes, methods, products, or apparatuses.

[0060] In order to solve the technical problems existing in the prior art, it is difficult for traditional single-modal models to fully explore the complementary information between different modalities, resulting in insufficient model generalization ability and inability to effectively cope with the growing multimodal data processing needs in practical applications. Multimodal learning methods are often unable to fully model the deep semantic relationship between images and texts, and the fusion strategy lacks flexibility and adaptability. Faced with complex scenarios such as medical diagnosis and remote sensing analysis, the model has poor robustness and the generalization ability is difficult to meet the needs of practical applications. The present invention provides a multimodal data fusion modeling method based on multi-task learning, such as Figure 1 As shown, a schematic diagram of a multimodal data fusion modeling method based on multi-task learning according to an embodiment of the present invention is shown. The multimodal data fusion modeling method based on multi-task learning includes:

[0061] Step S1, establishing a feature extraction unit, wherein the feature extraction unit includes a text encoder module and an image encoder module, wherein the image encoder module includes a DenseNet module for extracting image features from data; and the text encoder module includes a BERT module for extracting text features from data;

[0062] Step S2, fusing the image features extracted by the image encoder module and the text features extracted by the text encoder module to obtain first fused data;

[0063] Step S3: input the first fused data into the Transformer decoder and the weight layer respectively; the first fused data is used as the input of the Transformer decoder, and is passed through multiple self-attention layers and cross-attention layers in the Transformer decoder to obtain first processed data; the first fused data is input into at least two layers of the weight layers to obtain second processed data, and the second processed data is further fused with the first fused data to obtain third processed data; the first processed data, the third processed data, and the first fused data are input into the adaptive gating layer to obtain second fused data;

[0064] Step S4: input the fusion features in the second fusion data into the predicted label vector, and calculate the classification loss and average loss; calculate the contrast loss and average loss of the aligned text features and aligned image features in the second fusion data; if the classification loss, average loss and contrast loss all meet the preset conditions, output a multimodal data fusion model based on multi-task learning.

[0065] The present invention discloses a multimodal data fusion modeling method based on multi-task learning. First, the DenseNet module and the BERT module are used to extract the image and text features in the data respectively, and the first fused data is obtained after fusion; then the features are respectively input into the Transformer decoder and at least two weight layers for processing, and fused again with the original fused data through an adaptive gating layer to obtain the second fused data; finally, based on the second fused data, the classification loss and the contrast loss are calculated respectively. When both types of losses meet the preset conditions, the final multimodal data fusion model is output. The method first performs feature fusion preliminarily, and then combines the processing results of the Transformer decoder and the weight layer, dynamically adjusts the weights of each part of the information through the adaptive gating layer, strengthens the semantic alignment between the image and the text, effectively solves the cross-modal semantic problem, and improves the model's ability to understand complex semantic relationships. Through optimization such as classification loss, contrast loss, and average loss, the model can learn simultaneously in the classification task and the cross-modal alignment task, enhance the model's adaptability to different task scenarios, significantly improve the model's generalization ability and robustness in complex scenarios such as medical diagnosis and remote sensing analysis, and reduce data noise. Multimodal representation is constructed using image encoder, text encoder, decoder, fusion module and multi-task output. The global modeling capability of Transformer, the local modeling capability of ResNet and the adaptive gating mechanism are integrated to achieve efficient feature fusion and prediction in multi-task scenarios.

[0066] The image encoder module extracts discriminative feature representations from the original image data. In the preferred embodiment of the present invention, the image encoder is implemented based on the DenseNet-121 network structure. Based on the DenseNet-121 network structure, dense connections are introduced to improve feature reuse, alleviate the gradient vanishing problem, and effectively improve the learning ability of the model. Each layer of the DenseNet-121 network structure receives the feature maps of all previous layers as input, and has good information flow and parameter efficiency. After completing the DenseNet feature extraction, the extracted high-dimensional features are reduced and transformed through the fully connected layer so that the image feature map meets the consistency of the input dimension of the subsequent module. For example, assuming the input image is I∈R {B×3×K×W} , DenseNet extracts the feature f x =DenseNet(X)∈R {B×C} , transformed by the linear layer into the final image feature ImageFeature=Linear(f x )∈R {B×(C·D)} This module captures key information in images, including semantics, structure, and texture, laying a solid foundation for subsequent cross-modal fusion. B is the number of samples processed in parallel during each round of training or inference, C is the number of channels, D is the length of the feature dimension corresponding to each channel, K is the height of the image, and W is the width of the image.

[0067] The text encoder module is used to convert text data in natural language form into semantic vector representations. For example, the BERT model is used as the backbone network, making full use of its bidirectional attention mechanism to model the contextual information of the text.

[0068] In order to establish a deep interactive relationship between image and text features and realize cross-modal alignment and fusion of information, in a more preferred embodiment of the present invention, the first processed data, the third processed data, and the first fused data are input into the adaptive gating layer for weighted fusion. If the input is X:

[0069] X∈R {B×C×D}

[0070] Where R is the real number domain, B is the number of samples processed in parallel during each round of training or inference, C is the number of channels, corresponding to the number of feature channels of each modality before fusion, and D is the length of the feature dimension corresponding to each channel;

[0071] Input X first passes through the Transformer encoder to obtain the global context to obtain H;

[0072] H = TransformerEncoder(X)

[0073] Perform local enhancement through ResNet to obtain the output P of the ResNet module;

[0074] P=X+f(X)

[0075] Among them, f contains two layers of one-dimensional convolution, batch normalization and ReLU activation.

[0076] The Transformer encoder is used to model global context information; the ResNet module is used to enhance local feature learning, and the gating module implements adaptive weighted fusion of the two representations.

[0077] The H and P inputs are concatenated and input into the adaptive gating layer to calculate the adjustment weight r:

[0078] F=concat(P,H)

[0079] A=MLP(AdaptiveAvgPool1D(F))

[0080] r = Softmax(A)

[0081] The fused H′ is,

[0082] H′=P+r·H

[0083] Here, r controls the residual enhancement of the original residual information by the Transformer feature. This module has strong robustness and dynamic adaptability, effectively improving the learning effect of cross-modal representation.

[0084] The Transformer decoder is based on the Transformer architecture. It uses image features as input to the encoder and text features as input to the decoder, thereby constructing a joint image-text attention mechanism. Through multi-layer self-attention and cross-attention modules, the Transformer decoder enhances the mutual understanding between images and text in the semantic space. The positional encoding mechanism in the Transformer further helps the model capture feature ordering, thereby improving the accuracy of information matching. This module provides a unified, semantically rich feature representation for downstream fusion tasks and multi-task outputs.

[0085] In order to achieve efficient coordination of local and global modeling capabilities while maintaining a lightweight structure, in a more preferred embodiment of the present invention, the fused S is,

[0086] S=r·P+(1-r)·H′

[0087] Among them, r is used to control the dominant mode of the final output fusion and adjust the rebalancing of semantics and structure.

[0088] By combining the attention mechanism with the gating control in the gated recurrent unit, the learned weight parameters are used to dynamically adjust the transmission path and importance of the information flow. In multimodal fusion scenarios, the contributions of different modalities to specific tasks are often uneven and context-dependent. Traditional simple splicing or average fusion methods are unable to perceive this imbalance. The gating mechanism introduces adjustment weights that can automatically adjust the importance of each modal feature based on the current input content and task requirements.

[0089] The multi-output module is designed to handle the comparative learning of image and text features and the prediction requirements of the fusion task, respectively, thereby achieving the goal of multi-task learning. This design supports information cross-promotion between tasks of different modalities, improving model generalization and convergence efficiency. The contrastive loss aims to learn the semantic associations between images and text, bringing related image-text pairs closer in feature space and unrelated pairs further apart. By constraining the similarity between image and text features, the contrastive loss enables the model to learn more general and discriminative feature representations.

[0090] The classification loss directly measures the difference between the model prediction result and the true label, prompting the model to classify the input sample as accurately as possible. In order to minimize the classification loss, the model can learn the decision boundary between different categories, thereby improving the classification accuracy. In a more preferred embodiment of the present invention, in step S4, the classification loss is calculated as,

[0091]

[0092] Among them, N is the total number of samples, G is the number of label types, and y n,g is the true label value of the nth sample in class g, O n is the output vector of the nth sample;

[0093] The output matrix of image features is,

[0094] P∈R N×D

[0095] Among them, the image feature output matrix is ​​I∈R N×D , the output matrix of text features is T∈R N×D , normalize the two to get the similarity matrix S of image features and text features TI for,

[0096]

[0097] Among them, τ is the temperature parameter, which is used to adjust the smoothness of the distribution.

[0098] Target probability distribution matrix:

[0099]

[0100] Loss on the text side:

[0101]

[0102] The loss on the image side is:

[0103]

[0104] The contrast loss is,

[0105]

[0106] The total loss function after combining contrast loss and classification loss is,

[0107]

[0108] During training, the loss functions of each path are weighted and combined to form an overall loss function, supporting the collaborative optimization of multi-task objectives. Furthermore, the system supports a dynamic weight adjustment mechanism that adjusts the contribution of each subtask in real time based on task complexity and data distribution, further enhancing the model's stability and scalability in practical multimodal applications.

[0109] In order to make high-dimensional image features, such as the feature maps output by DenseNet, significantly reduce the number of parameters after dimensionality reduction, reduce the computational complexity of the model, and improve the training and reasoning speed, it is particularly suitable for large-scale data scenarios such as medical imaging and remote sensing. In a more preferred embodiment of the present invention, the image features extracted by the image encoder module are subjected to dimensionality reduction and transformation of the extracted high-dimensional features through a fully connected layer, so that the output image features are consistent with the input dimensions of the subsequent modules. The nonlinear activation function of the fully connected layer, such as ReLU, can further explore the implicit patterns in the image features, enhance the feature expression ability, and make the image features more in line with the requirements of downstream tasks. The dimensionality reduction process can be regarded as a feature compression, which helps to filter out redundant information and noise in high-dimensional features and focus on key semantic features.

[0110] Position encoding is used to inject position information into text tokens, compensating for the "position independence" defect of the Transformer model, enabling it to capture sequential dependencies in the text (such as subject-predicate structure) and improving the accuracy of semantic understanding. The input text is converted into a token sequence by the word segmenter, and after adding position encoding and type tags, it is input into the text encoder module for text feature extraction. Position encoding and type tags are the standard input format for pre-trained models such as BERT. This design can fully utilize the knowledge transfer capabilities of the pre-trained model and avoid performance degradation due to incompatible input formats.

[0111] In order to enable the multimodal model to more accurately capture the semantic relationship between images and texts and significantly improve the performance of cross-modal tasks, in a more preferred embodiment of the present invention, in step S3, the image feature is I∈R {B×C×D} , the text feature is T∈R {B×D} , the first fused data is input into the Transformer decoder respectively, and after fusion, it is Output = TransformerDecoder(I,T). The Transformer decoder is usually composed of multiple identical stacked layers. Each layer contains a self-attention layer (Self-Attention), a cross-attention layer (Cross-Attention), a feedforward neural network (FFN), layer normalization and residual connection. Residual connection and layer normalization are added after each sublayer. For example, the sequence features are compressed into a global vector. Gated fusion, that is, the output of the weight layer and the original fused data are dynamically weighted through an adaptive gating layer to achieve multi-path information integration.

[0112] The present invention also provides a system for the multi-modal data fusion modeling method based on multi-task learning as described above, the system comprising:

[0113] A feature extraction unit, the feature extraction unit comprising a text encoder module and an image encoder module, the image encoder module comprising a DenseNet module for extracting image features from the data; the text encoder module comprising a BERT module for extracting text features from the data;

[0114] a first fusion unit, configured to fuse the image features extracted by the image encoder module and the text features extracted by the text encoder module to obtain first fused data;

[0115] The second fusion unit is used to input the first fused data into the Transformer decoder and the weight layer respectively; the first fused data is used as the input of the Transformer decoder, and is passed through multiple self-attention layers and cross-attention layers in the Transformer decoder to obtain first processed data; the first fused data is respectively input into at least two layers of the weight layers to obtain second processed data, and the second processed data is further fused with the first fused data to obtain third processed data; the first processed data, the third processed data, and the first fused data are input into the adaptive gating layer to obtain second fused data;

[0116] The multi-task learning optimization unit is used to input the fusion features in the second fusion data into the predicted label vector and calculate the classification loss and average loss; calculate the contrast loss and average loss of the aligned text features and aligned image features in the second fusion data; if the classification loss, average loss and contrast loss all meet preset conditions, output a multimodal data fusion model based on multi-task learning.

[0117] As mentioned above, the system of the multimodal data fusion modeling method based on multi-task learning operates in coordination through four core units. First, the feature extraction unit uses the DenseNet module and the BERT module to extract image and text features respectively. The first fusion unit fuses the two types of features to form the first fusion data. The second fusion unit processes the first fusion data in different paths through the Transformer decoder and the weight layer, and integrates it with the adaptive gating layer to generate the second fusion data. The multi-task learning optimization unit calculates the classification loss, average loss and contrast loss based on the second fusion data. When each loss meets the preset conditions, the final multimodal data fusion model is output to achieve efficient processing and model optimization of cross-modal data. The cross-attention mechanism of the Transformer decoder is used to promote deep semantic alignment, the adaptive gating layer is used to dynamically screen key features, and the dual-path feature processing of the Transformer decoder and the weight layer is combined to take into account both global semantics and local details. The generalization ability of the model is improved through the joint optimization of classification and contrast losses. Finally, the global modeling ability of Transformer, the local modeling ability of ResNet and the adaptive gating mechanism are integrated to construct a multimodal representation including image encoder, text encoder, decoder, fusion module and multi-task output, realizing efficient feature fusion and accurate prediction of multiple tasks in complex scenarios.

[0118] In order to enable the model to take into account both global semantic coherence and local detail accuracy in complex tasks, effectively improve the integrity and robustness of multimodal feature representation, and reduce semantic deviation or detail loss problems caused by single feature modeling. In a more preferred case of the present invention, the second fusion unit also includes a fusion module based on a gating mechanism, and the fusion module based on the gating mechanism includes a Transformer encoder and a ResNet model, wherein the Transformer encoder is used for global context information modeling, and the ResNet model is used to enhance local feature learning. The long-distance semantic dependencies of cross-modal data are captured by the self-attention mechanism of the Transformer encoder, such as the association between the symptom description in the medical report and the global structure of the image, and the residual connection and hierarchical convolution of ResNet are used to enhance the extraction of local detail features, such as the edge texture of the lesion and the semantics of text keywords, and the weights of the two are dynamically balanced with the help of the gating mechanism.

[0119] For example, the present invention utilizes a publicly available multimodal medical dataset, the original images of which are derived from the "X-ray Lung Diseases Images" dataset published by Fernando Feltrin. The processed dataset contains 80,000 fully annotated chest X-ray images, covering the following eight categories of lung disease, with 10,000 images per category: obstructive lung disease, increased density, decreased density, chest structural changes, encapsulated lesions, degenerative infectious diseases, normal, and mediastinal changes. Each image is accompanied by a detailed clinical text description, providing corresponding symptoms and diagnostic information, forming a one-to-one correspondence between image and text, which can effectively support multimodal lung disease recognition and classification research based on X-ray images and clinical text.

[0120] By splitting the training and validation sets in an 8:2 ratio, after 200 training epochs, the accuracy rates reached 0.9999062485351333 and 0.9784361522595162, respectively. Compared to existing Vision Transformer and multimodal BERT-like models, this model introduces an innovative gating mechanism in its multimodal fusion mechanism, enabling dynamic weighted fusion of image and text features, effectively improving the model's ability to perceive the importance of information from different modalities. It not only automatically adjusts the fusion ratio of image and text based on context, enhancing the model's robustness to low-quality or modally incomplete samples, but also provides good interpretability, making the contribution of each modality in the decision-making process more controllable. Combining contrastive learning with classification supervision signals, the model not only possesses precise discriminative capabilities but also captures more robust and universal semantic representations in the feature space. Compared with the multimodal BERT model with static splicing or unified attention mechanism, this method significantly improves the migration ability and fusion flexibility in cross-domain tasks while maintaining the lightness of parameters, showing stronger practicality and promotion potential.

[0121] The present invention provides an electronic device, at least one processor; and

[0122] a memory communicatively connected to the at least one processor; wherein,

[0123] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned construction method.

[0124] The present invention provides a machine-readable storage medium having instructions stored thereon, wherein the instructions are used to enable a machine to execute the construction method of the present application as described above.

[0125] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0126] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0127] In the several embodiments provided herein, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units described is merely a logical functional division. In actual implementation, there may be other division methods, such as multiple units or components being combined or integrated into another system, or some features being ignored or not implemented. In addition, the coupling or direct coupling or communication connection shown or discussed may be through some interface, and the indirect coupling or communication connection of devices or units may be electrical or other forms.

[0128] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A multimodal data fusion modeling method based on multi-task learning, characterized in that: The multi-modal data fusion modeling method based on multi-task learning includes: Step S1, establishing a feature extraction unit, wherein the feature extraction unit includes a text encoder module and an image encoder module, wherein the image encoder module includes a DenseNet module for extracting image features from data; and the text encoder module includes a BERT module for extracting text features from data; Step S2, fusing the image features extracted by the image encoder module and the text features extracted by the text encoder module to obtain first fused data; Step S3: input the first fused data into the Transformer decoder and the weight layer respectively; the first fused data is used as the input of the Transformer decoder, and is passed through multiple self-attention layers and cross-attention layers in the Transformer decoder to obtain first processed data; the first fused data is input into at least two layers of the weight layers to obtain second processed data, and the second processed data is further fused with the first fused data to obtain third processed data; the first processed data, the third processed data, and the first fused data are input into the adaptive gating layer to obtain second fused data; Step S4: input the fusion features in the second fusion data into the predicted label vector, and calculate the classification loss and average loss; calculate the contrast loss and average loss of the aligned text features and aligned image features in the second fusion data; if the classification loss, average loss and contrast loss all meet the preset conditions, output a multimodal data fusion model based on multi-task learning.

2. The multimodal data fusion modeling method based on multi-task learning according to claim 1 is characterized in that: The first processed data, the third processed data, and the first fused data are input into the adaptive gating layer for weighted fusion. If the input is X: X∈R {B×C×D} Where R is the real number domain, B is the number of samples processed in parallel during each round of training or inference, C is the number of channels, corresponding to the number of feature channels of each modality before fusion, and D is the length of the feature dimension corresponding to each channel; Input X first passes through the Transformer encoder to obtain the global context to obtain H; H = TransformerEncoder(X) Perform local enhancement through ResNet to obtain the output P of the ResNet module; P=X+f(X) Among them, f contains two layers of one-dimensional convolution, batch normalization and ReLU activation; The H and P inputs are concatenated and input into the adaptive gating layer to calculate the adjustment weight r: F = concat(P, H) A=MLP(AdaptiveAvgPool1D(F)) r = Softmax(A) The fused H′ is, H′=P+r·H Among them, r is used to control the residual enhancement of the Transformer feature to the original residual information.

3. The multimodal data fusion modeling method based on multi-task learning according to claim 2 is characterized in that: The fused S is, S=r·P+(1-r)·H′ Among them, r is used to control the dominant mode of the final output fusion and adjust the rebalancing of semantics and structure.

4. The multimodal data fusion modeling method based on multi-task learning according to claim 1 is characterized in that: In step S4, the classification loss is calculated as, Among them, N is the total number of samples, G is the number of label types, and y n，g is the true label value of the nth sample in class g, O n is the output vector of the nth sample; The output matrix of image features is, P∈R N×D Among them, the image feature output matrix is ​​I∈R N×D , the output matrix of text features is T∈R N×D , normalize the two to get the similarity matrix S of image features and text features TI for, Among them, τ is the temperature parameter, which is used to adjust the smoothness of the distribution.

5. The multimodal data fusion modeling method based on multi-task learning according to any one of claims 1 to 4, characterized in that: The image features extracted by the image encoder module are subjected to dimensionality reduction and transformation on the extracted high-dimensional features through a fully connected layer, so that the output image features are consistent with the input dimensions of subsequent modules; The input text is converted into a token sequence by a word segmenter, and after adding position code and type tag, it is input into the text encoder module for text feature extraction.

6. The multimodal data fusion modeling method based on multi-task learning according to claim 5 is characterized in that: In step S3, the image feature is I∈R {B×C×D} , the text feature is T∈R {B×D} , the first fused data are respectively input into the Transformer decoder, and after fusion, the result is Output = TransformerDecoder (I, T).

7. A system for a multimodal data fusion modeling method based on multi-task learning according to any one of claims 1 to 6, characterized in that: The system comprises, A feature extraction unit, the feature extraction unit comprising a text encoder module and an image encoder module, the image encoder module comprising a DenseNet module for extracting image features from the data; the text encoder module comprising a BERT module for extracting text features from the data; a first fusion unit, configured to fuse the image features extracted by the image encoder module and the text features extracted by the text encoder module to obtain first fused data; The second fusion unit is used to input the first fused data into the Transformer decoder and the weight layer respectively; the first fused data is used as the input of the Transformer decoder, and is passed through multiple self-attention layers and cross-attention layers in the Transformer decoder to obtain first processed data; the first fused data is respectively input into at least two layers of the weight layers to obtain second processed data, and the second processed data is further fused with the first fused data to obtain third processed data; the first processed data, the third processed data, and the first fused data are input into the adaptive gating layer to obtain second fused data; The multi-task learning optimization unit is used to input the fusion features in the second fusion data into the predicted label vector and calculate the classification loss and average loss; calculate the contrast loss and average loss of the aligned text features and aligned image features in the second fusion data; if the classification loss, average loss and contrast loss all meet preset conditions, output a multimodal data fusion model based on multi-task learning.

8. The system according to claim 7, characterized in that The second fusion unit also includes a fusion module based on a gating mechanism, and the fusion module based on the gating mechanism includes a Transformer encoder and a ResNet model, wherein the Transformer encoder is used for global context information modeling and the ResNet model is used for enhancing local feature learning.

9. An electronic device, characterized in that: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal data fusion modeling method based on multi-task learning described in any one of claims 1-6.

10. A machine-readable storage medium, characterized in that The machine-readable storage medium stores instructions for enabling a machine to execute the multi-modal data fusion modeling method based on multi-task learning according to any one of claims 1 to 6 of the present invention.

Images