Multimodal pre-training method and system based on single-modal induction based on deep learning

Abstract

The present invention discloses a multimodal pre-training method and system based on single-modality induction based on deep learning. The work of the present invention aims to deal with the main challenges of multimodal image omics classification, namely the difficulty in extracting features from high-dimensional modal data, and the effective fusion considering high-order correlations. Specifically, the present invention first proposes a set of multi-head self-attention encoders to capture global structured features in long sequences. Then, the present invention designs a masked patch modeling paradigm, which masks random patch features from a fixed-length continuous subsequence of high-resolution images to capture latent features. Finally, the present invention combines the classification labels of paired patterns and proposes a triplet learning module to learn high-order correlations and extract global features. After pre-training, simple fine-tuning can be used to obtain classification results. The present invention can be used in visual question answering, image processing, autonomous driving, medical care and other fields.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and image processing, and specifically relates to a multimodal pre-training method and system based on single-modal induction based on deep learning. Background Art

[0002] Deep learning technology has revolutionized medical imaging, but designing a task-specific algorithm for pathology image-genomics multimodal analysis is challenging. (1) Pathology images with a resolution of 1 billion typically generate 15,000 foreground patches during preprocessing, making it difficult for attention-based network frameworks to extract accurate image-level representations. (2) Learning features from genomic data with tens of thousands of genes makes models such as Transformer unusable due to their quadratic computational complexity. (3) Feature fusion in image omics may not be able to model high-order correlations and the inherent structural characteristics of each modality, resulting in poor fusion effects. Most multimodal techniques are designed for applications such as chest X-rays with corresponding reports, CT and X-ray, CT and MRI, H&E cross-staining, etc., and are aligned through global features, local features, or multi-granularity. However, these works have not taken into account the challenges of processing high-resolution images and long sequence data.

[0003] Furthermore, visual language models in the computer vision community stand out for their remarkable versatility. However, due to limited computational resources, the most commonly used multimodal representation learning strategy, contrastive learning, relies on a large number of negative samples to avoid model collapse, which is unaffordable for images with a resolution of hundreds of billions. A huge domain heterogeneity also hinders their application to high-resolution images and long sequence data. Based on this multimodal fusion approach of high-resolution images and long sequence data, this paper proposes a new method, called unimodality-guided multimodal pre-training, to extract global image omics features. Summary of the Invention

[0004] To address the problem of difficulty in extracting global features from long sequences of unimodal data and high-resolution images, this paper proposes a unimodal-induced multimodal pre-training method and system based on deep learning. It introduces a universal encoder with structured learning capabilities and an effective unimodal-guided multimodal fusion module, which combines two pre-training objectives, triple learning and masked patch modeling, to achieve better performance.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] The multimodal pre-training method based on single-modal induction based on deep learning includes the following steps:

[0007] S1: Acquisition of long sequence data and high-resolution images;

[0008] S2: GroupMSA, a group-based multi-head self-attention encoder, obtains global structured features in long sequence data as unimodal features to induce multimodal pre-training;

[0009] S3: Based on the masked patch modeling pre-training strategy, random patch features are masked from a fixed-length continuous subsequence of the high-resolution image to obtain global structured features in the high-resolution image;

[0010] S4: Construct a unimodal induced triple learning module, input the global structured features in the long sequence data and the global structured features in the high-resolution image into the unimodal induced triple learning module, fuse them to obtain multimodal features, and then extract high-order global features of the multimodal features through CLS tag characters.

[0011] Preferably, in S2, the method for obtaining global structural features in long sequence data includes:

[0012] S21: Long sequence data Divided into N f non-overlapping segments;

[0013] S22: Based on the N f Non-overlapping segments, using linear mapping to obtain segment features Where d is the hidden dimension;

[0014] S23: Introduce an internal attention module to obtain the fragment features The local and global information in the fragment features The local and global information in is the global structural features in the long sequence data.

[0015] Preferably, in S23, an internal attention module is introduced to obtain the segment features Methods for local and global information in include:

[0016] S231: The segment features Divide into several groups, each group has N gr Learnable grouping labels; divide each group into several groups, each group generates (N f / N gr +1) mark;

[0017] S232: The (N f / N gr +1) tokens are fed into a multi-head self-attention block to extract intra-group information;

[0018] S233: Using local learning group labeling and final classification labeling CLS ge , modeling the cross-group interaction with another multi-head self-attention block on a global scale to obtain inter-group information, where the intra-group information and the inter-group information constitute the segment features Local and global information in .

[0019] Preferably, in S3, the method for obtaining global structural features in the high-resolution image includes:

[0020] S31: Setting a fixed threshold value of the continuous subsequence length L;

[0021] S32: Repeatedly construct small batches of continuous subsequences whose length L is less than a threshold, and set a masking ratio and a masking subscript set to obtain a mask sequence;

[0022] S33: A two-layer Nystrom-based patch aggregator and a lightweight reconstruction decoder are used to process the mask sequence, shield the random patch embeddings, and reconstruct the invisible information to obtain global structured features in the high-resolution image.

[0023] Preferably, in S4, a unimodal induced triple learning module is constructed, the global structured features in the long sequence data and the global structured features in the high-resolution image are input into the unimodal induced triple learning module, and the method of extracting high-order global features through CLS marker characters includes:

[0024] S41: pre-training the grouped multi-head self-attention encoder GroupMSA to obtain sequence features and freeze them in iterations;

[0025] S42: Learnable CLS for high-resolution images to label characters CLS img Add to the mask sequence to obtain patch embedding;

[0026] S43: Extract the patch embedding and the sequence features respectively, and add them to the final classification mark CLS ge Connect as CLS pat ;

[0027] S44: Using the triplet learning loss function to calculate the CLS pat Optimize

[0028] S45: Use a multilayer perceptron head to transform the CLS pat Mapping is performed to obtain the final high-order global features.

[0029] The present invention also provides a multimodal pre-training system based on single-modal induction of deep learning, comprising: an acquisition module, a first acquisition module, a second acquisition module and a pre-training module;

[0030] The acquisition module is used to acquire long sequence data and high-resolution images;

[0031] The first acquisition module is used for a group-based multi-head self-attention encoder GroupMSA to obtain global structured features in long sequence data as single-modal features for inducing multimodal pre-training;

[0032] The second acquisition module is based on a masked patch modeling pre-training strategy to mask random patch features from a fixed-length continuous subsequence to obtain global structural features in the high-resolution image;

[0033] The pre-training module is used to construct a unimodal induced triple learning module, input the global structured features in the long sequence data and the global structured features in the high-resolution image into the unimodal induced triple learning module, fuse them to obtain a multimodal image, and then extract the high-order global features of the multimodal image through CLS marker characters.

[0034] Preferably, the first obtaining module includes: a division unit, a segment feature obtaining unit and a first global feature obtaining unit;

[0035] The division unit is used to divide the long sequence data Divided into N f non-overlapping segments;

[0036] The segment feature obtaining unit is used to obtain the segment feature based on the N f Non-overlapping segments, using linear mapping to obtain segment features Where d is the hidden dimension;

[0037] The first global feature acquisition unit is used to introduce an internal attention module to obtain the segment feature The local and global information in the fragment features The local and global information in is the global structural features in the long sequence data.

[0038] Preferably, an internal attention module is introduced into the first global feature acquisition unit to obtain the segment feature The process of local and global information in includes:

[0039] The fragment features Divide into several groups, each group has N gr Learnable grouping labels; divide each group into several groups, each group generates (Nf / N gr +1) mark;

[0040] Each group generates (N f / N gr +1) tokens are fed into a multi-head self-attention block to extract intra-group information;

[0041] Leveraging locally learned group labels and final classification labels (CLS) ge , another multi-head self-attention cross-group interaction is used to model on a global scale to obtain inter-group information, where the intra-group information and the inter-group information constitute the segment features Local and global information in .

[0042] Preferably, the second obtaining module includes: a setting unit, a mask sequence obtaining unit and a second global feature obtaining unit;

[0043] The setting unit is used to set a fixed threshold value of the continuous subsequence length L;

[0044] The mask sequence obtaining unit is used to repeatedly construct small batches of continuous subsequences whose length L of the continuous subsequences is less than a threshold, and set a masking ratio and a masking subscript set to obtain a mask sequence;

[0045] The second global feature acquisition unit is used to process the mask sequence using a two-layer Nystrom-based patch aggregator and a lightweight reconstruction decoder, shield random patch embedding, and reconstruct invisible information to obtain global structured features in the high-resolution image.

[0046] Preferably, the pre-training module includes: a pre-training unit, an adding unit, a connecting unit, an optimizing unit and a high-order global feature obtaining unit;

[0047] The pre-training unit is used to pre-train the grouped multi-head self-attention encoder GroupMSA to obtain sequence features and freeze them in iterations;

[0048] The adding unit is used to add the learnable CLS tag character CLS of the high-resolution image img Add to the mask sequence to obtain patch embedding;

[0049] The connection unit is used to extract the patch embedding and the sequence features respectively, and the final classification mark CLS ge Connect as CLS pat ;

[0050] The optimization unit is used to optimize the CLS using the loss function of triplet learning pat Optimize

[0051] The high-order global feature acquisition unit is used to use a multi-layer perceptron head to transform the CLS pat Mapping is performed to obtain the final high-order global features.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] This paper first proposes a Transformer-based long sequence encoder, Grouped Multi-Head Self-Attention (GroupMSA), to capture global structured features in long sequence data. A pre-training paradigm, Masked Patch Modeling (MPM), is designed to mask random patch embeddings from a fixed-length contiguous subsequence of a high-resolution image. Assuming that a patch-level feature embedding can be reconstructed from its neighboring patches, our MPM only needs to recover the masked patch embeddings in a fixed-length subsequence, rather than processing all patches from the image. Furthermore, to learn the high-order correlations between the two modalities, we combine the CLS labels of multimodal data pairs to form a unified representation and propose a triplet learning module to distinguish global feature labels within a mini-batch. Notably, while our unified representation incorporates features from the entire long sequence data and partial images, thanks to the triplet learning module, we can still learn high-order correlations and discriminative global information between the two modalities before training. Furthermore, unlike self-supervised pre-training, our approach not only focuses on superior representation learning capabilities but also on class-dependent feature distributions, namely, intra- and inter-class variations. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 This is a flow chart of a multimodal pre-training method based on single-modality induction based on deep learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] Example 1

[0059] like Figure 1 As shown, given a multimodal dataset consisting of high-resolution image and long sequence data pairs (XI, XG), global feature representation is learned by completing masked patch modeling and triple learning modules. The overall framework consists of three parts: 1) group-based encoder GroupMSA, 2) efficient patch aggregator and 3) single-modality guided multimodal fusion.

[0060] In this embodiment, the present invention provides a multimodal pre-training method based on single-modal induction of deep learning, comprising the following steps:

[0061] S1: Collect long-sequence data and high-resolution images. Long-sequence data, such as genetic data, is complementary to high-resolution images (such as pathological images) and can better extract global features (genetic data + pathological images).

[0062] S2: GroupMSA, a multi-head self-attention encoder based on grouping, obtains global structured features in long sequence data;

[0063] S3: Based on the masked patch modeling pre-training strategy, random patch features are masked from a fixed-length continuous subsequence (i.e., a high-resolution image) to obtain global structured features in the high-resolution image;

[0064] S4: Construct a unimodal induced triple learning module, input the global structured features in long sequence data and the global structured features in high-resolution images into the unimodal induced triple learning module, fuse them to obtain multimodal features, and then extract the high-order global features of the multimodal features through CLS tag characters.

[0065] In this embodiment, the present invention proposes a Transformer-based long sequence encoder, Grouped Multi-Head Self-Attention (GroupMSA). GroupMSA is a specialized long sequence encoder for capturing structured features in long sequence data queues.

[0066] In S2, the method for obtaining global structured features in long sequence data based on the grouped multi-head self-attention encoder GroupMSA includes:

[0067] S21: Long sequence data Divided into N f non-overlapping segments;

[0068] S22: Based on N f Non-overlapping segments, using linear mapping to obtain segment features Where d is the hidden dimension;

[0069] S23: Introduce an internal attention module to obtain segment features The local and global information in The local and global information in is the global structured features in long sequence data.

[0070] In this embodiment, in S23, an internal attention module is introduced to obtain the segment features Methods for local and global information in include:

[0071] S231: The fragment features Divide into several groups, each group has N gr Learnable grouping labels; divide each group into several groups, each group generates (N f / N gr +1) mark;

[0072] S232: The (N f / N gr +1) tokens are fed into a multi-head self-attention block to extract intra-group information;

[0073] S233: Leveraging locally learned group labels and final classification labels CLS ge , another multi-head self-attention block is used to model the cross-group interaction on a global scale to obtain inter-group information, where the intra-group information and inter-group information constitute the segment features That is, GroupMSA can be used to learn semantic information from long sequence data queues.

[0074] In this embodiment, in S3, the method for obtaining global structural features in the high-resolution image includes:

[0075] S31: Setting a fixed threshold value of the continuous subsequence length L;

[0076] S32: Repeatedly construct small batches of continuous subsequences whose length L is less than a threshold, and set the masking ratio and masking subscript set to obtain a mask sequence;

[0077] S33: A two-layer Nystrom-based patch aggregator and a lightweight reconstruction decoder are used to process mask sequences, shield random patch embeddings, and reconstruct invisible information to obtain global structured features in high-resolution images.

[0078] Specifically, the present invention utilizes a masked patch modeling (MPM) pre-training strategy that trains continuous subsequences of fixed length. We mask random patch embeddings in

[15] and reconstruct invisible information. Based on experience, we set the fixed subsequence length L to 6000, and repeatedly construct mini-batches of sequences smaller than L, set the masking ratio to 50%, and record the masked subscript set as M∈R 0.5L Then, a two-layer Nystrom-based patch aggregator and a lightweight reconstruction decoder are used to process the mask sequence H MPM , the reconstructed sequence is expressed as Please note that this paper reconstructs the missing features instead of the original resolution of the masked area, which is different from traditional MIM methods such as SimMIM and MAE. Reconstruction L1 loss calculation method: Among them, 1[] is the indicator function.

[0079] In classification tasks, the Transformer-based architecture requires CLS markers to extract accurate global information, which is more difficult due to the challenges of long sequences. In addition, in order to build a small batch high-resolution processing, the subsequences intercepted in the pre-training stage of the MPM may not be sufficient to represent image-level features. To overcome these problems, the present invention further proposes a unimodal induced triple learning module, which uses high-resolution images and long sequence data as input and extracts high-order global features through CLS markers.

[0080] In S4, a unimodal induced triple learning module is constructed, global structured features in long sequence data and global structured features in high-resolution images are input into the unimodal induced triple learning module, and a method for extracting high-order global features through CLS labeled characters includes:

[0081] S41: Pre-train the grouped multi-head self-attention encoder GroupMSA (the pre-training process is: using the locally learned group label and the final classification label CLS ge , modeling cross-group interactions with another multi-head self-attention block at the global scale), obtaining sequential features (features of long sequence data) and freezing them in iterations;

[0082] S42: Learnable CLS for high-resolution images to label characters CLS img Add to the mask sequence to obtain patch embedding (features of high-resolution image);

[0083] S43: Extract the input patch embedding and sequence features respectively, and add them to the final classification mark CLS ge Connect as CLS pat ;

[0084] S44: Using triplet learning loss function for CLS pat Optimize

[0085] S45: Using Multilayer Perceptron Head to CLS pat Mapping is performed to obtain the final high-order global features.

[0086] Specifically, assume that the present invention obtains a triple list {x, x+, x-} in the current iteration, where x, x+, x- are the anchor points CLS pat , positive CLS pat and negative CLS pat In order to improve the global modeling ability, that is, to extract more accurate global level features, it is expected that the distance between the anchor point and the positive sample is closer, while the distance between the negative samples is farther. The loss function calculation method for optimizing triplet learning is:

[0087]

[0088] δ represents a threshold, for example, δ = 0.8. Finally, the loss function of GiMP pre-training is: Lpre = Ltri + Lrec, and a multi-layer perceptron (MLP) head is used to transform CLS pat Mapped to the final global feature.

[0089] Example 2

[0090] The present invention also provides a multimodal pre-training system based on single-modal induction of deep learning, comprising: an acquisition module, a first acquisition module, a second acquisition module and a pre-training module;

[0091] The acquisition module is used to collect long-sequence data and high-resolution images. Long-sequence data includes, for example, genetic data, which is complementary to high-resolution images (such as pathological images) and can better capture global features (genetic data + pathological images).

[0092] The first acquisition module is used for the group-based multi-head self-attention encoder GroupMSA to obtain global structured features in long sequence data as unimodal features for inducing multimodal pre-training;

[0093] The second acquisition module is based on the masked patch modeling pre-training strategy, which masks random patch features from a fixed-length continuous subsequence of the high-resolution image (i.e., the high-resolution image) to obtain the global structural features in the high-resolution image;

[0094] The pre-training module is used to construct a unimodal induced triple learning module. The global structured features in the long sequence data and the global structured features in the high-resolution image are input into the unimodal induced triple learning module to fuse them to obtain a multimodal image. The high-order global features of the multimodal image are then extracted through CLS tag characters.

[0095] In this embodiment, the first obtaining module includes: a division unit, a segment feature obtaining unit and a first global feature obtaining unit;

[0096] The division unit is used to divide long sequence data into Divided into N f non-overlapping segments;

[0097] The segment feature acquisition unit is used to obtain the segment feature based on N f Non-overlapping segments, using linear mapping to obtain segment features Where d is the hidden dimension;

[0098] The first global feature acquisition unit is used to introduce an internal attention module to obtain segment features The local and global information in The local and global information in is the global structured features in long sequence data.

[0099] In this embodiment, an internal attention module is introduced into the first global feature acquisition unit to obtain the segment feature The process of local and global information in includes:

[0100] The fragment features Divide into several groups, each group has N gr Learnable grouping labels; divide each group into several groups, each group generates (N f / N gr +1) mark;

[0101] The (N f / N gr +1) tokens are fed into a multi-head self-attention block to extract intra-group information;

[0102] Leveraging locally learned group labels and final classification labels (CLS) ge , another multi-head self-attention block is used to model the cross-group interaction on a global scale to obtain inter-group information, where the intra-group information and inter-group information constitute the segment features Local and global information in .

[0103] In this embodiment, the second obtaining module includes: a setting unit, a mask sequence obtaining unit, and a second global feature obtaining unit;

[0104] The setting unit is used to set a threshold value of a fixed continuous subsequence length L;

[0105] The mask sequence obtaining unit is used to repeatedly construct small batches of continuous subsequences whose length L is less than a threshold, and set a masking ratio and a masking subscript set to obtain a mask sequence;

[0106] The second global feature acquisition unit is used to process the mask sequence using a two-layer Nystrom-based patch aggregator and a lightweight reconstruction decoder, shield the random patch embedding, and reconstruct the invisible information to obtain global structured features in the high-resolution image.

[0107] In this embodiment, the pre-training module includes: a pre-training unit, an adding unit, a connecting unit, an optimizing unit and a high-order global feature obtaining unit;

[0108] The pre-training unit is used to pre-train the grouped multi-head self-attention encoder GroupMSA (the pre-training process is: using the locally learned group label and the final classification label CLS ge , modeling cross-group interactions with another multi-head self-attention block at the global scale), obtaining sequential features (characteristics of long sequence data), and freezing in iterations;

[0109] Add a unit for learning CLS to label characters of high-resolution images img Add to the mask sequence to obtain patch embedding (features of high-resolution image);

[0110] The connection unit is used to extract the input patch embedding and sequence features respectively, and the final classification mark CLS ge Connect as CLS pat ;

[0111] The optimization unit is used to optimize CLS using the triplet learning loss function pat Optimize

[0112] The high-order global feature acquisition unit is used to convert the CLS into pat Mapping is performed to obtain the final high-order global features.

[0113] The main feature of this paper is a novel multimodal pre-training method to address the difficulty in extracting global features from long sequences of unimodal data and high-resolution images. Specifically, the paper introduces a universal encoder with structured learning capabilities and an efficient unimodal-guided multimodal fusion module that combines two pre-training objectives: triplet learning and masked patch modeling. This method achieves superior performance compared to other state-of-the-art methods.

[0114] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A multimodal pre-training method based on single-modal induction based on deep learning, characterized in that: The following steps are involved: S1: Acquisition of long sequence data and high-resolution images; S2: GroupMSA, a group-based multi-head self-attention encoder, obtains global structured features in long sequence data as unimodal features to induce multimodal pre-training; S3: Based on the masked patch modeling pre-training strategy, random patch features are masked from a fixed-length continuous subsequence of the high-resolution image to obtain global structured features in the high-resolution image; S4: constructing a unimodal induced triple learning module, inputting the global structured features in the long sequence data and the global structured features in the high-resolution image into the unimodal induced triple learning module, fusing them to obtain multimodal features, and then extracting high-order global features of the multimodal features through CLS marker characters; In S2, the method for obtaining global structural features in long sequence data includes: S21: Long sequence data Divided into non-overlapping segments; S22: Based on the Non-overlapping segments, using linear mapping to obtain segment features , where d is the hidden dimension; S23: Introduce an internal attention module to obtain the fragment features The local and global information in the fragment features The local and global information in is the global structural features in the long sequence data; In S3, the method for obtaining global structural features in the high-resolution image includes: S31: Setting a fixed threshold value of the continuous subsequence length L; S32: Repeatedly construct small batches of continuous subsequences whose length L is less than a threshold, and set a masking ratio and a masking subscript set to obtain a mask sequence; S33: Processing the mask sequence with a two-layer Nystrom-based patch aggregator and a lightweight reconstruction decoder to mask random patch embeddings and reconstruct invisible information to obtain global structured features in the high-resolution image; In S4, a unimodal induced triple learning module is constructed, the global structured features in the long sequence data and the global structured features in the high-resolution image are input into the unimodal induced triple learning module, and the method of extracting high-order global features through CLS tag characters includes: S41: pre-training the grouped multi-head self-attention encoder GroupMSA to obtain sequence features and freeze them in iterations; S42: Learnable CLS character tagging for high-resolution images Add to the mask sequence to obtain patch embedding; S43: Extract the patch embedding and the sequence features respectively, and add them to the final classification mark Connect as ; S44: Using the triplet learning loss function to Optimize S45: Use a multi-layer perceptron head to Mapping is performed to obtain the final high-order global features.

2. The multimodal pre-training method based on single-modal induction based on deep learning according to claim 1, characterized in that: In S23, an internal attention module is introduced to obtain the segment features Methods for local and global information in include: S231: The segment features Divided into several groups, each group has A learnable grouping tag; each group is divided into several groups, each group generates Marks; S232: Each group generates The tokens are fed into a multi-head self-attention block to extract the information within the group; S233: Utilizing locally learned group labels and final classification labels , modeling the cross-group interaction with another multi-head self-attention block on a global scale to obtain inter-group information, where the intra-group information and the inter-group information constitute the segment features Local and global information in .

3. A multimodal pre-training system based on single-modal induction based on deep learning, characterized by: include: An acquisition module, a first acquisition module, a second acquisition module, and a pre-training module; The acquisition module is used to acquire long sequence data and high-resolution images; The first acquisition module is used for a group-based multi-head self-attention encoder GroupMSA to obtain global structured features in long sequence data as single-modal features for inducing multimodal pre-training; The second acquisition module is based on a masked patch modeling pre-training strategy, which masks random patch features from a fixed-length continuous subsequence of the high-resolution image to obtain global structural features in the high-resolution image; The pre-training module is used to construct a unimodal induced triple learning module, input the global structured features in the long sequence data and the global structured features in the high-resolution image into the unimodal induced triple learning module, fuse them to obtain multimodal features, and then extract high-order global features of the multimodal features through CLS marker characters; The first obtaining module includes: a segmentation unit, a segment feature obtaining unit and a first global feature obtaining unit; The division unit is used to divide the long sequence data Divided into non-overlapping segments; The segment feature obtaining unit is used to obtain the segment feature based on the Non-overlapping segments, using linear mapping to obtain segment features , where d is the hidden dimension; The first global feature acquisition unit is used to introduce an internal attention module to obtain the segment feature The local and global information in the fragment features The local and global information in is the global structural features in the long sequence data; The second obtaining module includes: a setting unit, a mask sequence obtaining unit and a second global feature obtaining unit; The setting unit is used to set a fixed threshold value of the continuous subsequence length L; The mask sequence obtaining unit is used to repeatedly construct small batches of continuous subsequences whose length L of the continuous subsequences is less than a threshold, and set a masking ratio and a masking subscript set to obtain a mask sequence; The second global feature acquisition unit is used to process the mask sequence using a two-layer Nystrom-based patch aggregator and a lightweight reconstruction decoder, mask random patch embedding, and reconstruct invisible information to obtain global structured features in the high-resolution image; The pre-training module includes: a pre-training unit, an adding unit, a connecting unit, an optimizing unit and a high-order global feature obtaining unit; The pre-training unit is used to pre-train the grouped multi-head self-attention encoder GroupMSA to obtain sequence features and freeze them in iterations; The adding unit is used to add learnable CLS labeled characters to the high-resolution image Add to the mask sequence to obtain patch embedding; The connection unit is used to extract the patch embedding and the sequence features respectively, and the final classification label Connect as ; The optimization unit is used to optimize the Optimize The high-order global feature acquisition unit is used to use a multi-layer perceptron head to Mapping is performed to obtain the final high-order global features.

4. The multimodal pre-training system based on single-modal induction based on deep learning according to claim 3, characterized in that: In the first global feature acquisition unit, an internal attention module is introduced to obtain the segment feature The process of local and global information in includes: The fragment features Divided into several groups, each group has A learnable grouping tag; each group is divided into several groups, each group generates Marks; Each group produces The tokens are fed into a multi-head self-attention block to extract the information within the group; Leverage locally learned group labels and final classification labels , another multi-head self-attention cross-group interaction is used to model on a global scale to obtain inter-group information, where the intra-group information and the inter-group information constitute the segment features Local and global information in .

Images