A large-scale model pre-training method based on multi-information perception

Abstract

The present invention provides a large-scale model pre-training method based on multivariate information perception, inputs the collected image, text, and audio data into a pre-training database, annotates the pre-training database in small batches to form a recognition database, and uses the data in the pre-training database to perform unsupervised mode pre-training on the recognition model. The pre-training is carried out from three perception states respectively, and the parameters obtained by the pre-training model are used as the initialization parameters of the recognition model. The supervised learning training of the model is performed based on the recognition database to obtain the final intelligent recognition model. The present invention takes unsupervised learning as the cornerstone, relies on the powerful ability of large-scale model pre-training, and uses discriminant models and generative models to perform large-scale model pre-training, thereby realizing the effective use of massive unlabeled data; supports the input of a wide range of multimodal data, and adopts mutual information maximization-contrast loss as the core optimization goal, so that the model can smoothly process the complex relationship between multimodal data.

Description

Technical Field

[0001] The present invention relates to the general field of image data processing or generation technology, and in particular to a large-scale model pre-training method based on multivariate information perception. Background Art

[0002] When building an artificial intelligence recognition system, it is necessary to collect a large amount of target data, and then accurately annotate the collected data by domain experts or relevant annotators to build a huge database. The annotation of data in natural scenes can be distributed to the data annotation team, and a large number of annotators will annotate the data. Due to the large amount of data, the cost of manual annotation is high, and it is easy to make annotation errors during annotation.

[0003] In the field of medical imaging, data labeling requires a large number of experienced and senior experts and scholars to accurately determine the specific categories of data. However, the medical industry, which is short of professional talents, finds it difficult to carry out large-scale data labeling with limited labeling resources. Modern deep learning technology based on supervised learning requires a large amount of labeled data to drive the learning of models with practical application value.

[0004] In nature and medicine, large amounts of unlabeled data are generated every day, including data in different sensory states such as images, voice, and text. If the large amount of unlabeled data can be fully utilized to assist in the training of intelligent recognition models, the problem of neural networks requiring large amounts of labeled data can be greatly alleviated. At the same time, the model can obtain information from a more comprehensive and diverse perspective, and can achieve better performance with a small amount of labeled data.

[0005] Therefore, a model training method that fully utilizes unlabeled data is needed. Summary of the invention

[0006] The present invention aims to solve the problem of excessive data annotation and provide a large-scale model pre-training method based on multivariate information perception. It makes full use of data of different perception states constantly generated in nature and medicine, and cooperates with a small amount of annotated data to enable the model to achieve optimal recognition performance. The present invention only needs to annotate a small amount of data, greatly reducing the required annotation resources, and actively avoids the generation of "category imbalance" problems in the data collection stage, thereby constructing an intelligent recognition model with faster iteration speed, better recognition performance, and stronger generalization ability. This method has extremely high practical application value.

[0007] The existing technology indiscriminately labels all collected data as much as possible, and the demand for data labeling is high; a statistical learning recognition model based on supervised learning is constructed, including the construction of feature processes, data preprocessing, and training of statistical learning models; a deep learning recognition model based on supervised learning is constructed, and feature extraction and automatic classification are directly performed by a deep neural network; the model is trained using only data from a single perception state, and data from other perception states other than the image is not utilized, which inhibits the improvement of model performance.

[0008] Existing methods treat all data in the training dataset equally and label as much as possible, which relies heavily on a large amount of labeling resources. In addition, there is a large amount of redundant information in the labeling results, which causes a waste of labeling resources. They ignore a large amount of unlabeled data, which can provide different levels of information for the same sample from different perception states, which helps to improve the recognition performance of the model. Existing methods are based on labeled data-driven and rely heavily on the labeling progress and quality of the data, which greatly prolongs the training cycle of the model and makes it impossible to iterate the model efficiently and quickly.

[0009] The present invention provides a large-scale model pre-training method based on multivariate information perception, comprising the following steps:

[0010] S1. Collect medical image data, medical text data, medical audio data and natural data to form a pre-training database, where the data in the pre-training database is unlabeled data; select representative small batch samples from the pre-training database for labeling to form a recognition database;

[0011] S2. Build an image-aware pre-training model. The image-aware pre-training model uses image restoration as a proxy task and can restore the original image that is randomly occluded. The image-aware pre-training model pre-trains the image data in the pre-training database in an unsupervised mode and uses a mean square error loss function to restore the target image.

[0012] S3. Build a text / audio-aware pre-training model. The text / audio-aware pre-training model is a language model that can use the context information of the text to predict the covered text. The text / audio-aware pre-training model pre-trains the text data and audio data in the pre-training database in an unsupervised mode and uses a cross-entropy loss function to restore the target text.

[0013] S4, using the pre-training parameters of the image perception state pre-training model and the pre-training parameters of the text / audio perception state pre-training model as initialization parameters of the multi-information interaction module, and performing supervised learning training of the multi-information interaction module based on the recognition database, so that the multi-information interaction module correctly matches the image perception state and the text perception state of the input image to complete the recognition task;

[0014] The supervised learning training method of the multivariate information interaction module is as follows: the input image with text description in the recognition database is processed into blocks to obtain block images, the block images and texts are respectively subjected to feature extraction by the image perception state pre-training model and the text / audio perception state pre-training model, and the extracted image features and text features are both subjected to the MLP mapping layer with shared parameters to obtain mapped image features and mapped text features; the mapped image features and mapped text features are interacted in pairs by the multivariate perception interaction module, so that the mapped image features of each image block interact with all the mapped text features, and features are extracted from each other to obtain image and text features that integrate multivariate information;

[0015] When the contrast loss function for matching the image perception state and the text perception state reaches the optimization target, the training of a large-scale model based on multi-information perception is completed.

[0016] The present invention provides a large-scale model pre-training method based on multivariate information perception. As a preferred method, in step S1, medical image data is acquired through an electron microscope, a phase contrast microscope and an optical microscope and collected using a microscope camera;

[0017] Natural data refers to data in natural scenes, including natural image data, natural text data, natural audio data and natural video data, which are collected through cameras or networks;

[0018] Medical text data is obtained from the medical knowledge base, and medical audio data is obtained from the medical video knowledge base and the medical audio knowledge base, and audio information is extracted and converted into text.

[0019] The present invention provides a large-scale model pre-training method based on multivariate information perception. As a preferred method, when constructing a recognition database, doctors use annotation tools to annotate medical image data, medical text data, and medical audio data with category information, and use LabelMe annotation tools to annotate natural data.

[0020] The labeled data in the recognition database is divided into training set, validation set and test set according to the proportion.

[0021] The present invention provides a large-scale model pre-training method based on multivariate information perception. As a preferred embodiment, in step S2, the image perception state pre-training model includes an image perception encoding structure and an image perception decoding structure;

[0022] The image perception coding structure divides the input image into blocks to obtain at least 16*16 image blocks, then blocks more than 50% of the image blocks to obtain blocked image blocks, retains the unblocked image blocks, rearranges the unblocked image blocks into a new block map and inputs it into the neural network for encoding and feature extraction;

[0023] The image perception decoding structure receives the features output by the image perception encoding structure and the learnable representation of the occluded image block and outputs the complete original image; each occluded image block shares the same learnable representation, each learnable representation introduces position information to indicate the position of the occluded image block in the original image, and the learnable representation fuses the information of the remaining unoccluded image blocks to restore the occluded image blocks and the original image content; the number of parameters of the image perception decoding structure is smaller than that of the image perception encoding structure;

[0024] The image-aware pre-trained model uses the mean square error loss function to restore the target image:

[0025] ;

[0026] Among them, M is the number of occluded image blocks in each input image, N is the number of images in the dataset, y is the prediction result of the neural network, t is the actual original image content, i is the i-th sample, and j is the j-th block or word in the i-th sample.

[0027] The present invention provides a large-scale model pre-training method based on multivariate information perception. As a preferred method, in the image perception coding structure, each image is divided into 16*16 image blocks, and then 75% of the image blocks are blocked;

[0028] In the image perception decoding structure, the learnable "representation" of the occluded image block and the features output by the image perception encoding structure are sorted in the order of the original image, input into the image perception decoding structure, and the original image content is restored by the neural network;

[0029] The number of parameters in the image perception decoding structure is only 10% of that in the image perception coding structure.

[0030] The present invention provides a large-scale model pre-training method based on multivariate information perception. As a preferred method, in step S3, the audio data is extracted and converted into text information, which is processed together with the text data by the text / audio perception state pre-training model;

[0031] The text / audio perception pre-training model includes a text / audio perception encoding structure and a text / audio perception decoding structure;

[0032] The text / audio-aware encoding structure receives the original text and outputs the encoded text features;

[0033] The processing method of the text / audio-aware coding structure is as follows: the original text is segmented by a word segmentation tool, and then the sentence after the word segmentation is encoded by word vectorization and converted into a feature representation of a word vector. Then, the words in the sentence are randomly masked at a ratio of less than 30%, and the representation of the masked words is replaced by a learnable masked representation; at the same time, the representation of the masked words is randomly replaced with the representation of other words in the entire vocabulary. After the replacement is completed, the masked sentence is feature extracted and output;

[0034] The text / audio perception decoding structure receives the features output by the text / audio perception encoding structure, including the features after the representation of the masked words is mapped, and outputs a complete sentence; the representation of the masked words interacts with the context of the text, and the masked words are predicted based on the context of the text; the neural network analyzes the logic of the text, understands the grammatical structure and semantic information of the text, and predicts the masked words;

[0035] The text / audio-aware pre-trained model predicts the masked words using a cross-entropy loss function:

[0036] ;

[0037] Among them, m is the number of occluded words in each sentence, n is the number of texts in the dataset, q is the prediction result of the neural network, and p is the real occluded word.

[0038] The present invention provides a large-scale model pre-training method based on multi-information perception. As a preferred method, the features output by the text / audio perception coding structure include grammatical structure information, high-order semantic information and logical information of the text;

[0039] The text / audio perceptual coding structure randomly masks words in a sentence at a rate of 15%.

[0040] The present invention provides a large-scale model pre-training method based on multi-information perception. As a preferred embodiment, in step S4, the multi-information interaction module matches the image perception state and the text perception state through a contrast loss function;

[0041] ;

[0042] Where K is the number of text / image samples, is the image feature of the current sample, is the text feature of the current sample.

[0043] The present invention provides a large-scale model pre-training method based on multi-information perception. As a preferred method, in step S4, the annotation results of the medical image data in the recognition database are textualized so that each medical image data has a text matching the image annotation result.

[0044] The present invention provides a large-scale model pre-training method based on multivariate information perception, which, as a preferred embodiment, further includes step S5:

[0045] S5. After the large-scale model pre-training is completed, the data set that needs to be classified and the category set that needs to be identified are input into the large-scale model to match the image data with the text data. The category represented by the best matched text is the specific category of the image. Finally, the specific category of each image is counted, and the statistical results are formed and output to complete the intelligent recognition task.

[0046] The present invention has the following advantages:

[0047] (1) The present invention makes full use of unlabeled image data, text data, audio data and other data of different perception states generated every day in nature and the medical field to complete the pre-training of a large model that integrates multiple information, thereby assisting in improving the performance of the intelligent recognition model.

[0048] (2) The present invention only needs to label a small batch of data to surpass the performance of the model trained with large-scale labeled data, alleviating the problem of shortage of labeling resources, significantly reducing the labeling cost, and accelerating the update and iteration of the model.

[0049] (3) Fewer labeled data make cross-labeling possible, greatly reducing the possibility of data labeling errors. The labeled errors of the selected training data sets are smaller and cleaner. This effectively reduces the model error, making the model optimization faster and the performance better.

[0050] (4) The present invention makes full use of data of different perceptual states of the same sample, such as images, texts, and voices, to make up for the insufficiency of information obtained by each perceptual state, making the features extracted by the neural network more comprehensive and complete, and strongly improving the recognition performance of the intelligent recognition model. The recognition model finally obtained has high application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of a large-scale model pre-training method based on multi-information perception;

[0052] Figure 2 A schematic diagram of data annotation for a large-scale model pre-training method based on multi-information perception;

[0053] Figure 3a A sample diagram of a large-scale model pre-training method based on multi-information perception Figure 1 ;

[0054] Figure 3b A sample diagram of a large-scale model pre-training method based on multi-information perception Figure 2 ;

[0055] Figure 3c A sample schematic diagram of a large-scale model pre-training method based on multi-information perception 3;

[0056] Figure 3d A sample diagram of a large-scale model pre-training method based on multi-information perception Figure 4 ;

[0057] Figure 3e A sample diagram of a large-scale model pre-training method based on multi-information perception Figure 5 ;

[0058] Figure 3f A sample diagram of a large-scale model pre-training method based on multi-information perception Figure 6 ;

[0059] Figure 4 A schematic diagram of the encoding structure of a large-scale model pre-training method based on multi-information perception;

[0060] Figure 5 A schematic diagram of the specific medical annotation content and effects of a large-scale model pre-training method based on multi-information perception;

[0061] Figure 6 A schematic diagram of the encoding structure of a large-scale model pre-training method based on multi-information perception;

[0062] Figure 7 This is a multi-sensing interaction flow chart of a large-scale model pre-training method based on multi-information perception;

[0063] Figure 8 The flowchart of the algorithm operation is a large-scale model pre-training method based on multi-information perception. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Example 1

[0065] like Figure 1 As shown, a large-scale model pre-training method based on multi-source information perception is adopted in the present invention. The large-scale model pre-training technology based on "multi-source information perception" is adopted.

[0066] 1. Data Collection

[0067] Different types of data have different collection methods. For medical image data, images can be acquired with the help of electron microscopes, phase contrast microscopes, and optical microscopes, and data can be collected using microscope cameras. For natural image data, data can be collected using ordinary cameras or the Internet. For text data, crawler technology can be used to crawl information in relevant knowledge bases. For voice data, crawler technology can be used to crawl information in relevant video and audio knowledge bases, and audio information can be extracted and converted into text.

[0068] 2. Data annotation

[0069] Only small batches of data need to be labeled. For medical data, professional doctors can use specific labeling tools to label categories and other information. For natural data, experts in related fields or outsourced service teams can use labeling tools such as LabelMe to label data. Figure 2 When the amount of annotation reaches a certain scale, the annotated data is divided into training set, validation set and test set according to a certain ratio to prepare data for model building.

[0070] 3. Model construction

[0071] Large-scale model pre-training method based on "multiple information perception".

[0072] The system mainly consists of three parts

[0073] 1. Collect data of different perception states, including images, text, audio and other data, clean them and merge them into the "pre-training" training database. Filter the data in the "pre-training" database to obtain small batches of data as data to be labeled, and have professional labelers label the data to form a "recognition" database.

[0074] 2. Use the data in the "pre-training" database to perform unsupervised "pre-training" on the recognition model. The pre-training is carried out from three perception states, namely image, text, and audio perception states.

[0075] 3. Use the parameters obtained from the "pre-training" model in 2 as the initialization parameters of the recognition model, perform supervised learning training on the model based on the "recognition" database, and obtain the final intelligent recognition model.

[0076] The present invention constructs an intelligent recognition system based on the large-scale model pre-training technology of "multi-information perception".

[0077] First, collect data on a large scale, including data in three modalities: images, text, and audio, to build a "pre-training database". Secondly, select a small batch of representative samples based on the constructed "pre-training" database and have them labeled by professional labelers to build a "recognition" database. Then, build a "multi-information perception" pre-training model, and pre-train the model based on the constructed "pre-training" database. Finally, use the parameters obtained from the model pre-training as the initialization parameters of the recognition model, and train it based on the constructed "recognition" database to obtain the final intelligent recognition model. The intelligent recognition model outputs each sample category and forms statistical results.

[0078] (1) Building a training database

[0079] First, collect relevant data on a large scale, collect samples under different perception states and conditions, and enrich the diversity of data. In the data collection stage, collect data from all scenarios as much as possible, covering various samples that may be encountered in actual application scenarios, such as Figure 3a~3f As shown in the figure, a huge "pre-training" database is constructed. At the same time, when constructing the "recognition" database, it is not only helpful to select samples with the largest amount of information of different attributes and build a diverse recognition database, but also helps to improve the classification and generalization capabilities of the recognition model, making the model have strong practical value.

[0080] (2) Building an image perception pre-training model

[0081] In order to train the model based on unlabeled image data, the present invention proposes a pre-trained model with image restoration as the proxy task. The model randomly blocks a large area of ​​the original image, requiring the pre-trained model to completely restore the image content of the blocked part.

[0082] Image data is rich in low-level semantic information, and there is a serious redundancy relationship between a large amount of information. General pre-training proxy tasks cannot eliminate a large amount of redundant information, and the semantic information in the features extracted by the neural network is relatively vague. The present invention adopts large-scale image occlusion technology to eliminate redundant information over a large area, and forces the model to use the remaining non-redundant image blocks for image restoration. This method can completely extract the high-order semantic information of the image, which helps to improve the performance of the downstream recognition model.

[0083] This module consists of two parts: encoding structure and decoding structure.

[0084] A. Coding structure

[0085] like Figure 4As shown in , the structure receives an image as input and outputs encoded image features. First, the image is "blocked", and each image is divided into small blocks of 16*16 to form a block image. Block the block image by 75%, and retain the remaining 25% of the block image after blocking. The remaining 25% of the block images are rearranged to form a new block image, which is input into the coding architecture composed of a neural network for encoding. The highly sparse input image suppresses the ability of the neural network to simply restore the original image content through similar adjacent pixel information, while improving the training speed of the model, creating conditions for designing an efficient coding structure. Only unblocked block images are input into the coding structure for feature extraction. As shown in Figure 5 As shown in the figure, the specific annotation content and effect diagram of medicine are matched with the corresponding text description. The model can learn the feature description corresponding to each cell image, understand the image content and correctly identify the specific category.

[0086] B. Decoding structure

[0087] like Figure 6 As shown, the structure receives the features output by the encoding structure and the learnable "representation" of the occluded image block, and outputs the complete original image. Each occluded image block shares the same learnable "representation", and each learnable "representation" introduces corresponding position information, indicating the position of the occluded image block in the original image. The learnable "representation" integrates the information of the remaining unoccluded image blocks, restores the occluded image blocks, and restores the original image content. The learnable "representation" of the occluded image block and the features output by the encoding structure are sorted in sequence according to the order in the original image, input into the decoding structure, and the original image content is restored by the neural network. At this time, the input size of the decoding structure is the same as the original image, and the amount of calculation becomes larger. The present invention simplifies the design of the decoding structure, and its parameter amount is only 10% of the encoding structure. Although the input becomes larger, the overall amount of calculation remains unchanged, the accuracy remains unchanged, and the iteration efficiency of the model is greatly improved.

[0088] The present invention selects the mean square error loss function to restore the target image, and the specific form is as follows:

[0089] ;

[0090] Where M represents the number of occluded image blocks in each image, and N represents the number of images in the dataset. y represents the prediction result of the neural network, t represents the true original image content, i represents the i-th image sample, and j represents the j-th image block in the i-th image sample.

[0091] (3) Construction of text / audio perception pre-training model

[0092] Audio information can be converted into text information, so the present invention extracts the audio information, converts it into text information and processes it together with the text information.

[0093] In daily life, a large amount of text / audio information is generated every day. This naturally provides high-quality material for neural networks to understand human language and text. Based on the sequentiality and high-order semantic properties of text information, the present invention uses a language model to cover some words in a sentence, and uses the contextual information of the text to predict the covered text, so that the neural network can understand the semantic information and grammatical rules of the text.

[0094] A. Coding structure

[0095] This structure receives the original text as input and outputs the encoded text features. First, the text is segmented using a word segmentation tool so that the complete sentence can be correctly divided. Then, the sentence after word segmentation is encoded by word vectorization and converted into a feature representation of the word vector. Subsequently, the words in the sentence are randomly masked at a rate of 15%, and the representation of the masked words is replaced by a learnable "masked representation". At the same time, in order to prevent the neural network from being overly sensitive to the "masked representation", the representation of the masked words is randomly replaced with the representation of other words in the entire vocabulary. After the replacement is completed, the masked sentence is input into the encoding structure for feature extraction and output.

[0096] B. Decoding structure

[0097] This structure receives the features output by the encoding structure, including the features after the representation of the occluded words is mapped, and outputs a complete sentence. The representation of the occluded words interacts with the context of the text, and the occluded words are predicted based on the text context. In this process, the neural network needs to analyze the logic of the text and understand the grammatical structure and semantic information of the text in order to correctly predict the occluded words. The features output by the encoding structure contain grammatical structure information, high-order semantic information, and logical information of the text, making it possible for the model to understand the text, and a large amount of unlabeled text can be fully utilized to assist model training.

[0098] The present invention selects the cross entropy loss function to predict the masked words, and the specific form is as follows:

[0099] ;

[0100] Among them, M is the number of obscured words in each sentence, N is the number of texts in the dataset, q is the prediction result of the neural network, and p is the real word that is obscured.

[0101] (4) Construction of multi-information interaction module

[0102] After pre-training of images and text / audio with large-scale data, image features and text features rich in high-order semantic information are extracted. At this time, small batches of data can be used to fine-tune the above features to facilitate the extraction of task-related features and adapt to downstream tasks. When constructing this module, the present invention has small batches of annotated image data and corresponding categories. At the same time, the present invention textualizes the annotation results of the image, that is, if the original annotation is cat, the textualized result is "This is a cat playing." Each image has its own specific text description, which is used to match the image and text.

[0103] A sample can be described by different perceptual states. Different perceptual states focus on different aspects of the sample. For example, the image state focuses on the low-level information of the sample, such as texture, color, etc. The text / audio state focuses on the high-level information of the image, such as the abstract summary of the sample or the individual's cognition of the image. If the data of different perceptual states fully interact, they can complement each other's strengths and weaknesses, supplement the information that each perceptual state cannot directly obtain, and assist the model to improve recognition performance.

[0104] like Figure 7 As shown, the present invention provides a cross-sensory state interaction network, in which different perceptual states can fully interact with each other, each taking what is needed, and making full use of the information of different perceptual states of the same sample. First, the input image is processed into blocks to obtain block images. The block images and texts are respectively subjected to feature extraction through the image perceptual state module and the text perceptual state modality. The extracted image and text features are passed through the MLP mapping layer with shared parameters to obtain the mapped image and text features. The mapped image and text features are interacted in pairs through the multi-sensory interaction module, and each image block feature interacts with all text features, extracting related features from each other to obtain image and text features that integrate multi-information. The present invention hopes that the distance between the image and text features of the same sample is as small as possible, and the distance between the text and image features of different samples is as large as possible, so that the image perceptual state and text perceptual state of each sample can be correctly matched, thereby completing the image intelligent recognition task. For example, if an image of a cat is input, five candidate categories are given, namely, cat, dog, lion, tiger, and elephant. The intelligent recognition model can correctly match the image perception state of the cat and the text perception state of "This is an image of a cat", thereby completing the recognition task and outputting the image category and related statistical results.

[0105] The contrast loss function matches the image perception state with the text perception state, and the specific form is as follows:

[0106] ;

[0107] Where K is the number of text / image samples, is the image feature of the current sample, is the text feature of the current sample.

[0108] Through the optimization of the above loss function, the model can correctly match the image and the corresponding text to achieve the purpose of image recognition.

[0109] (5) Category identification and output of statistical results

[0110] The present invention adopts the strategy of image and text matching to perform image recognition tasks. Given a data set that needs to be classified, given a set of categories that need to be identified, such as rod-shaped granulocytes, lobed granulocytes, lymphocytes, and monocytes in blood cell morphology identification. The text information corresponding to each category is converted into complete sentences such as "This is an image of a lymphocyte" and "This is an image of a lobed granulocyte". Then the image data can be matched with the text data, and the category represented by the best text matched is the specific category of the image. Finally, the specific category of each image is counted, and the statistical results are formed and output to complete the intelligent recognition task.

[0111] A set of categories such as cats, dogs, elephants, lions, and tigers can also be given. The text information corresponding to each category is converted into complete sentences such as "This is a photo of a lion" and "This is an image of an elephant". The image data can then be matched with the text data, and the category represented by the best matched text is the specific category of the image. Finally, the specific category of each image is counted, the statistical results are formed and output, and the intelligent recognition task is completed. The algorithm operation process is as follows: Figure 8 shown.

[0112] This paper takes unsupervised learning as the cornerstone and relies on the powerful ability of large-scale model pre-training. It uses discriminative models and generative models to perform large-scale model pre-training, and realizes the effective use of massive unlabeled data. This strategy not only greatly reduces the burden of data annotation, but also makes full use of the stability and generalization ability of large models in complex scenarios.

[0113] The present invention supports the input of a wide range of multimodal data, including but not limited to text, images, videos, and audio, greatly broadening the scope of application and practicality of the model. At the same time, in order to achieve deep fusion and efficient interaction of multimodal features, the present invention innovatively introduces an "interactive attention mechanism", which can intelligently identify and enhance important information associations between different modalities, surpassing simple feature splicing and achieving true cross-modal feature fusion.

[0114] In addition, the present invention abandons the traditional classifier structure and adopts "mutual information maximization-contrast loss" as the core optimization goal. It guides model learning by maximizing the mutual information between different modal data of the same sample, and directly outputs the results based on the size of the mutual information. This innovation simplifies the model architecture and fundamentally changes the optimization goal, enabling the model to smoothly process the complex relationship between multimodal data, showing technical advantages and application prospects that are completely different from traditional supervised learning methods.

[0115] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A large-scale model pre-training method based on multivariate information perception, characterized in that: The following steps are involved: S1. Collect medical image data, medical text data, medical audio data and natural data to form a pre-training database, where the data in the pre-training database is unlabeled data; select representative small batch samples from the pre-training database for labeling to form a recognition database; S2. Building an image-aware pre-training model, wherein the image-aware pre-training model uses image restoration as a proxy task to restore the randomly occluded original image. The image-aware pre-training model performs unsupervised pre-training on the image data in the pre-training database and uses a mean square error loss function to restore the target image until the image-aware pre-training model completely restores the image content of the occluded part; S3, building a text / audio-aware pre-training model, wherein the text / audio-aware pre-training model is a language model, and uses the context information of the text to predict the covered text. The text / audio-aware pre-training model performs unsupervised pre-training on the text data and audio data in the pre-training database and uses a cross entropy loss function to restore the target text; S4, using the pre-training parameters of the image perception state pre-training model and the pre-training parameters of the text / audio perception state pre-training model as initialization parameters of a multi-information interaction module, and performing supervised learning training of the multi-information interaction module based on the recognition database, so that the multi-information interaction module correctly matches the image perception state and the text perception state of the input image to complete the recognition task; The supervised learning training method of the multi-information interaction module is as follows: the input image with text description in the recognition database is processed into blocks to obtain block images, the block images and texts are respectively subjected to feature extraction by the image perception state pre-training model and the text / audio perception state pre-training model, and the extracted image features and text features are subjected to the MLP mapping layer with shared parameters to obtain mapped image features and mapped text features; the mapped image features and the mapped text features are interacted in pairs in the multi-information interaction module through the interactive attention mechanism, so that the mapped image features of each image block interact with all mapped text features, and features are extracted from each other to obtain image and text features that integrate multi-information; When the contrast loss function for matching the image perception state and the text perception state reaches the optimization target, a large-scale model training based on multi-information perception is completed; S5. After the pre-training of the large-scale model is completed, the text information in the category set that needs to be identified is converted into complete sentences, and the data set that needs to be classified and the complete sentences of the category set that needs to be identified are input into the large-scale model to match the image data with the text data. The image state that focuses on the low-order information of the sample and the text / audio state that focuses on the high-order information of the image fully interact and supplement the information that cannot be directly obtained by each perception state. The category represented by the best matched text is the specific category of the image. Finally, the specific category of each image is counted, and the statistical results are formed and output to complete the intelligent recognition task.

2. The large-scale model pre-training method based on multivariate information perception according to claim 1, characterized in that: In step S1, the medical image data is acquired through an electron microscope, a phase contrast microscope and an optical microscope and collected using a microscope camera; The natural data is data in a natural scene, including natural image data, natural text data, natural audio data and natural video data, and the data is collected through a camera or a network; The medical text data is obtained from a medical knowledge base, and the medical audio data is obtained from a medical video knowledge base and a medical audio knowledge base, and audio information is extracted and converted into text.

3. The large-scale model pre-training method based on multivariate information perception according to claim 2, characterized in that: When constructing the recognition database, the doctor uses a labeling tool to label the medical image data, the medical text data, and the medical audio data with category information, and uses a LabelMe labeling tool to label the natural data; The labeled data in the recognition database is divided into a training set, a validation set and a test set according to proportion.

4. The large-scale model pre-training method based on multivariate information perception according to claim 1, characterized in that: In step S2, the image perception state pre-training model includes an image perception encoding structure and an image perception decoding structure; The image perception coding structure divides the input image into blocks to obtain 16*16 image blocks, then blocks more than 50% of the image blocks to obtain blocked image blocks, retains unblocked image blocks, rearranges the unblocked image blocks to form a new block map and inputs the new block map into a neural network for coding and feature extraction; The image perception decoding structure receives the features output by the image perception encoding structure and the learnable representation of the occluded image block and outputs a complete original image; each of the occluded image blocks shares the same learnable representation, each learnable representation introduces position information to indicate the position of the occluded image block in the original image, and the learnable representation fuses the information of the remaining unoccluded image blocks to restore the occluded image blocks and recover the original image content; the parameter amount of the image perception decoding structure is less than the parameter amount of the image perception encoding structure; The image-aware pre-trained model uses a mean square error loss function to restore the target image: ； Among them, M is the number of occluded image blocks in each input image, N is the number of images in the dataset, y is the prediction result of the neural network, t is the actual original image content, i is the i-th sample, and j is the j-th block or word in the i-th sample.

5. The large-scale model pre-training method based on multivariate information perception according to claim 4, characterized in that: In the image perception coding structure, each image is divided into 16*16 image blocks, and then 75% of the image blocks are blocked; In the image perception decoding structure, the learnable "representation" of the occluded image block and the features output by the image perception encoding structure are sequentially sorted according to the order in the original image, input into the image perception decoding structure, and the original image content is restored by the neural network; The number of parameters of the image perceptual decoding structure is only 10% of the number of parameters of the image perceptual coding structure.

6. The large-scale model pre-training method based on multivariate information perception according to claim 1, characterized in that: In step S3, the audio data is extracted and converted into text information, which is processed together with the text data by the text / audio perception state pre-trained model; The text / audio perception pre-training model includes a text / audio perception encoding structure and a text / audio perception decoding structure; The text / audio perception encoding structure receives the original text and outputs the encoded text features; The processing method of the text / audio perception coding structure is as follows: the original text is segmented by a word segmentation tool, and then the sentence after the word segmentation is encoded by word vectorization and converted into a feature representation of a word vector, and then the words in the sentence are randomly masked at a ratio less than 30%, and the representation of the masked words is replaced by a learnable masked representation; at the same time, the representation of the masked words is randomly replaced with the representation of other words in the entire vocabulary, and after the replacement is completed, the masked sentence is feature extracted and output; The text / audio perception decoding structure receives the features output by the text / audio perception encoding structure, including the features after the representation of the masked words is mapped, and outputs a complete sentence; the representation of the masked words interacts with the context of the text, and the masked words are predicted based on the text context; The neural network analyzes the logic of the text, understands the grammatical structure and semantic information of the text, and predicts the obscured words; The text / audio-aware pre-trained model predicts masked words using a cross-entropy loss function: ； Among them, m is the number of occluded words in each sentence, n is the number of texts in the dataset, q is the prediction result of the neural network, and p is the real occluded word.

7. The large-scale model pre-training method based on multivariate information perception according to claim 6, characterized in that: The features output by the text / audio perceptual coding structure include grammatical structure information, high-order semantic information and logical information of the text; The text / audio perceptual coding structure randomly masks words in a sentence at a rate of 15%.

8. The large-scale model pre-training method based on multivariate information perception according to claim 1, characterized in that: In step S4, the multivariate information interaction module matches the image perception state and the text perception state through a contrast loss function; ； Where K is the number of text / image samples, is the image feature of the current sample, is the text feature of the current sample.

9. The large-scale model pre-training method based on multivariate information perception according to claim 1, characterized in that: In step S4, the annotation results of the medical image data in the recognition database are converted into text, so that each medical image data has text matching the image annotation results.

Images