Multi-modal information fusion AI platform and application method

Abstract

The invention relates to a multi-modal information fusion AI platform and an application method, and the platform comprises a second data input module which is used for allowing a medical institution to access and input data; the second data management module is used for checking and preprocessing the data input by the second data input module; the downstream task fine tuning module is used for inheriting model parameters output by the base model pre-training module in the base model pre-training technical scheme and performing fine tuning on the model to obtain a reasoning model; and the prediction result output module is used for testing the performance of the reasoning model and displaying a prediction result. The method can cope with more complex and changeable clinical scenes, and high dependence of traditional supervised learning on data is reduced. The method has the capability of multi-modal feature fusion. The fusion is not only beneficial to capturing more comprehensive information, but also can improve the recognition ability of the model to lesions. The multi-task processing capability enables the model to cope with various clinical requirements at the same time.

Description

Technical Field

[0001] The present invention relates to the cross-technical field of artificial intelligence and medical diagnosis, and specifically to an AI platform and application method for multimodal information fusion. Background Art

[0002] As more and more Internet companies and research teams join the AIGC (Artificial Intelligence Generative Content) wave, a large number of open-source large language models are launched. All sectors of society are aware of the opportunities brought by this artificial intelligence revolution and have joined the research of large models. Since 1994, at the first Computational Medicine, Public Health and Biotechnology Conference held in Austin, USA, computational medicine has emerged and become an important frontier research direction in medicine. In 2012, Raymond Winslow, a professor of biomedical engineering at Johns Hopkins University in the United States, pointed out in a review article entitled "Computational Medicine: From Model to Clinic" that computational medicine has moved from theory to practice. It can be seen that with the continuous advancement of artificial intelligence technology, the cross-integration of medicine and computer science has also entered a new stage. Medical large models not only rely on massive medical imaging data, but also combine the powerful ability of deep learning to dig out hidden medical knowledge and laws in complex data relationships. These models can provide doctors with more accurate and comprehensive diagnostic support and treatment recommendations through the fusion analysis of multimodal data such as medical images and clinical records.

[0003] A base model refers to a pre-trained model, on which a secondary training and fine-tuning can be performed for specific downstream tasks to adapt to tasks in different scenarios. Base models have been widely used in different fields: First, in the field of natural language processing, the GPT (Generative Pre-Training) model adapts to different natural language processing tasks by performing unsupervised pre-training on a large-scale text corpus, and then achieves excellent text generation, machine translation, dialogue system and other functions with the help of fine-tuning; the LLaMA model uses text instruction fine-tuning, and its feature is that it uses high-quality question-answering supervision data provided by ChatGPT. Second, in the field of computer vision, the MAE (‌Masked Autoencoders) model proves that the model pre-trained by self-supervised learning also has good generalization in the field of vision. Third, in the field of multimodal fusion, the CLIP (Contrastive Language-Image Pre-Training) model breaks the fixed category label paradigm and allows the model to learn the matching relationship between text and image through contrastive learning; the LLaVA model is the first large model that attempts to expand instruction fine-tuning from text to the multimodal field of text plus vision.

[0004] Among the existing AI platforms, adaptability is limited and they are usually only good at processing single-modality data: medical images and medical texts. The model structure used by these platforms is relatively simple, lacking feature extraction modules for different modalities and mechanisms for integrating features from different modalities. These models may encounter problems of gradient vanishing or gradient exploding as the number of layers increases. The models of these platforms may not be as effective as other methods in processing global information. The training effects for specific tasks (segmentation, classification, detection) rely on a large amount of high-quality labeled data. Summary of the invention

[0005] The multimodal information fusion AI platform proposed in the present invention is intended to solve the problems of limited adaptability of existing AI platforms, the problem of being able to process only a single modality, and the problem that model training relies on a large amount of high-quality labeled data.

[0006] To achieve the above object, the present invention provides the following technical solutions: An AI platform for multimodal information fusion, comprising: A second data input module, used to allow a medical institution to access and input data, wherein the data includes text data and image data, wherein the text data is a diagnosis report, and the image data is 3D medical imaging data; A second data governance module, used for reviewing and preprocessing the data input by the second data input module; The downstream task fine-tuning module is used to inherit the model parameters output by the base model pre-training module in the technical solution of base model pre-training, and fine-tune the model to obtain an inference model; The prediction result output module is used to test the performance of the inference model and display the prediction results.

[0007] As a further technical solution of the present invention, the technical solution of base model pre-training includes: The first data input module is used to receive data provided by the scientific research platform; The first data governance module is used to control data quality and pre-process data; The base model pre-training module will use the preprocessed data to train a base model and its corresponding parameters for 3D medical imaging data to connect to different downstream tasks.

[0008] As a further technical solution of the present invention, the scientific research platform in the first data input module is an image data scientific research platform.

[0009] As a further technical solution of the present invention, the first data governance module includes: a data quality control unit, a standard text data processing unit and an image data processing unit that use human-machine combination to efficiently and accurately process data; Among them, the data quality control unit is controlled by combining human and machine for high efficiency and accuracy: low-quality data is screened through manual annotation process, annotated data with gold standard is obtained, and the anatomical or pathological structure is assisted by using automatic segmentation tools based on SAM; Standard text data processing unit: For the text data of the diagnosis report, the standard token vectorization steps include cleaning useless characters, text segmentation, removing stop words, and vectorization; Image data processing unit: For 3D medical imaging data, the accuracy of the model is improved and I / O conflicts are reduced through data screening, packaging of large files, offline data preprocessing and enhancement. For data screening rules, corresponding screening rules are specified for each body part in the form of a whitelist to ensure that the data that passes the screening is basically directly available data. For offline data preprocessing, the 3D medical imaging data stored in the nii format is first loaded, and the data is converted into data in the numpy array format after preprocessing operations such as clip and resize. The array data of the specified number n is spliced ​​into an array of shape (n, 1, 256, 256, 32) or (n, 1, 256, 256, 128), and finally saved as large files of about 5G stored in the h5 format. After testing, it was found that after packaging 1200 NII files into the H5 format, the total size was reduced from 12.6GB to 4.7GB, which effectively reduced memory usage.

[0010] As a further technical solution of the present invention, the base model pre-training module constructs a similarity matrix of text-image feature vector pairs by inputting the data processed by the first data management module into the text encoder and the image encoder in sequence; wherein, N text data are output token text vectors through a text encoder that inherits the LLaMA3 parameters, and N 3D image data are output image vectors through a 3D image encoder based on LLaVA-v1.5; the token text vectors and image vectors are used to construct a similarity matrix (calculated by cosine similarity), wherein each element of the similarity matrix represents the inner product of the corresponding image and text vectors, and during the training process, the inner products on the diagonal represent matching image-text pairs, while the inner products at other positions represent unmatched image-text pairs.

[0011] As a further technical solution of the present invention, in the base model pre-training module, for the modeling of image-text matching, we set the InfoNCE loss function to make the learning goal of the model clear: maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs; in order to prevent the model from overfitting during training, we introduced the SiLU activation function, the Dropout layer and the L2 regularization technology, which can effectively enhance the generalization ability of the model; In terms of training strategy, we first adopt a phased training method to improve training efficiency and model stability. Specifically, we first inherit the pre-trained LLaMA3 text encoder, then train the 3D image encoder based on LLaVA-v1.5 separately, and finally perform joint training of image and text; in addition, we use mixed precision training to accelerate the training process while ensuring the accuracy of the model; use a warm up strategy to gradually increase the learning rate to avoid the initial excessive learning rate affecting the model convergence; and use a cosine annealing strategy to dynamically adjust the learning rate, so as to better control the learning rate changes during training and promote efficient convergence of the model; Finally, the training ends and the convergence is judged: the model will stop training when the number of epochs we specified in advance is reached; in addition, to save computing resources, we introduce an early stop strategy; the model performance is monitored in real time on the validation set, and the training is terminated early when the loss difference between two adjacent epochs is less than 0.0001, thereby avoiding unnecessary training time and possible overfitting.

[0012] As a further technical solution of the present invention, the downstream task fine-tuning module will first inherit the model parameters output by the base model pre-training module; then, the model is fine-tuned using the data passed through the second data governance module; instead of fine-tuning all parameters, we only train the decoder: a text decoder based on LLaMA3, an image decoder based on UNETR, and an image decoder based on MLP classification head; finally, after completing the model training, the base model weights of the base model pre-training module are merged with the fine-tuning model weights of the downstream task fine-tuning module to obtain an inference model.

[0013] As a further technical solution of the present invention, the prediction result output module is used to make predictions for the input test set data, and then output the prediction results of the case report, the segmentation results of the lesions and anatomy of the medical image, and the positive and negative classification results; The AI ​​platform will use 3D visualization of nii images to show the difference between the gold standard and the predicted results, and will display the actual results and predicted results of the case report next to the image. If there is no real result data, only the predicted results will be displayed on the platform for users to refer to the output results of the model.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. Zero-shot learning capability. Although existing AI medical imaging diagnostic platforms can predict and output case report diagnoses, their prediction results are relatively rigid, and their outputs are labels of fixed categories. For example, if the labels used in model training are lung nodules and tuberculosis, then the model cannot predict the lesion of pulmonary fibrosis, and can only output negative results (no lung nodules and no tuberculosis), which is obviously not in line with reality. However, the present invention can achieve zero-shot results because it uses the idea of ​​the CLIP model. With the help of the powerful learning ability of the large model - the feature extraction ability of LLaMA3 and LLaVA-v1.5, the present invention obtains deep information hidden in case reports and medical imaging data by improving the generalization of the model, and then outputs words that the model has not seen (pulmonary fibrosis). This can not only cope with more complex and changeable clinical scenarios, but also reduce the high dependence of traditional supervised learning on data.

[0015] 2. Capability of multimodal feature fusion. The present invention realizes the fusion of multimodal features by processing text (case reports) and image (medical imaging) data simultaneously. This fusion not only helps to capture more comprehensive information, but also improves the model's ability to identify lesions.

[0016] 3. Ability to connect to different downstream tasks. In the inference stage, the present invention achieves multi-tasking by connecting to different decoders. This includes outputting the prediction results of case reports, the segmentation results of lesions and anatomy of medical images, and the positive and negative classification results. This multi-tasking capability enables the model to respond to multiple clinical needs at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of an application method of an AI platform for multimodal information fusion provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0019] The core of the multimodal information fusion platform implemented by the present invention is the training of medical imaging models enabled by multimodal models. The present invention aims to realize the complete process of a third party that lacks machine learning knowledge logging into the platform, inputting its own data and adjusting its labels for consistency, then training and predicting multimodal information fusion, and finally visualizing the results. The technical solution needs to be divided into two sub-solutions: base model pre-training and downstream task fine-tuning. The base model pre-training includes a first data input module, a first data governance module, and a base model pre-training module, and its implementation and execution will be completed within the company. This solution will eventually give an interface for the implementation of the training module in the technical solution of downstream task fine-tuning, which is the main function of the present invention. The technical solution for downstream task fine-tuning is divided into five modules, including a second data input module, a second data governance module, a downstream task fine-tuning module, and a prediction result output module. The implementation and execution of all modules in the technical solution of downstream task fine-tuning will be completed on an AI platform for multimodal information fusion proposed by the present invention.

[0020] The first data input module is used to receive data provided by the scientific research platform, and the data covers five parts: CT-abdomen, CT-brain, CT-chest, MRA-brain, and MR-brain. The data includes text data and image data. The text data is a diagnosis report, and the image data is 3D medical imaging data. The scientific research platform in the first data input module is an imaging data scientific research platform, which can be specifically the big data scientific research platform of Yimai Sunshine Medical Imaging Center, which has nearly 10 million cases of data.

[0021] The first data governance module is used to control data quality and pre-process data. In terms of data quality control, since all data comes from the big data research platform of the Yimai Sunshine Medical Imaging Center, the writing of diagnostic reports follows almost similar filling specifications, and the annotation of medical imaging data follows exactly the same core definition table. This is the main measure to ensure data consistency and is also the unique advantage of the data. In terms of pre-processing data, the huge amount of data determines the special pre-processing method of offline pre-processing. Specifically: In this embodiment, the first data governance module includes: a data quality control unit that uses human-machine combination to perform efficient and accurate data processing, a standard text data processing unit, and an image data processing unit; Among them, the data quality control unit is controlled by combining human and machine for high efficiency and accuracy: low-quality data is screened through manual annotation process, annotated data with gold standard is obtained, and the anatomical or pathological structure is assisted by using automatic segmentation tools based on SAM; Standard text data processing unit: For the text data of the diagnostic report, the standard token vectorization steps include cleaning useless characters, text segmentation, removing stop words, and vectorization; Image data processing unit: For 3D medical imaging data, the accuracy of the model is improved and I / O conflicts are reduced through data screening, packaging of large files, offline data preprocessing and enhancement. For data screening rules, corresponding screening rules are specified for each body part in the form of a whitelist to ensure that the data that passes the screening is basically directly available data. For offline data preprocessing, the 3D medical imaging data stored in the nii format is first loaded, and the data is converted into data in the numpy array format after preprocessing operations such as clip and resize. The array data of the specified number n is spliced ​​into an array of shape (n, 1, 256, 256, 32) or (n, 1, 256, 256, 128), and finally saved as large files of about 5G stored in the h5 format. After testing, it was found that after packaging 1200 NII files into the H5 format, the total size was reduced from 12.6GB to 4.7GB, which effectively reduced memory usage.

[0022] The base model pre-training module will use the pre-processed data to train a base model and its corresponding parameters for 3D medical imaging data to connect to different downstream tasks; In this embodiment, the base model pre-training module constructs a similarity matrix of text-image feature vector pairs by inputting the data processed by the first data management module into the text encoder and the image encoder in sequence; wherein, N text data are output token text vectors through a text encoder that inherits the LLaMA3 parameters, and N 3D image data are output image vectors through a 3D image encoder based on LLaVA-v1.5; the token text vectors and image vectors are used to construct a similarity matrix (calculated by cosine similarity), wherein each element of the similarity matrix represents the inner product of the corresponding image and text vectors, and during the training process, the inner products on the diagonal represent matching image-text pairs, while the inner products at other positions represent unmatched image-text pairs.

[0023] In the base model pre-training module, for the modeling of image-text matching, we set the InfoNCE loss function to make the learning goal of the model clear: maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs; in order to prevent the model from overfitting during training, we introduced the SiLU activation function, Dropout layer and L2 regularization technology, which can effectively enhance the generalization ability of the model; In terms of training strategy, we first adopt a phased training method to improve training efficiency and model stability. Specifically, we first inherit the pre-trained LLaMA3 text encoder, then train the 3D image encoder based on LLaVA-v1.5 separately, and finally perform joint training of image and text; in addition, we use mixed precision training to accelerate the training process while ensuring the accuracy of the model; use a warm up strategy to gradually increase the learning rate to avoid the initial excessive learning rate affecting the model convergence; and use a cosine annealing strategy to dynamically adjust the learning rate, so as to better control the learning rate changes during training and promote efficient convergence of the model; Finally, the training ends and the convergence is judged: the model will stop training when the number of epochs we specified in advance is reached; in addition, to save computing resources, we introduce an early stop strategy; the model performance is monitored in real time on the validation set, and the training is terminated early when the loss difference between two adjacent epochs is less than 0.0001, thereby avoiding unnecessary training time and possible overfitting.

[0024] The second data input module is used to allow medical institutions to access and input data; the downstream task fine-tuning solution uses medical imaging data with diagnostic reports from more than 100 cooperative medical institutions to fine-tune the model and output results. First, in order to ensure the consistency of data from different medical structures, we will communicate with the algorithms and annotation teams of the cooperative medical institutions in advance to ensure that the same core definition table is used for medical imaging data annotation and the same filling specifications are used for writing diagnostic reports.

[0025] Secondly, multiple cooperative medical institutions can access the module and input data at the same time without the possibility of conflict. Finally, for data input operations, the quantity and quality of input data are controlled by each medical institution. This module only provides a data input interface and is not directly connected to the hospital information terminal to prevent privacy issues such as data leakage.

[0026] The second data governance module is used to review and preprocess the data input by the second data input module; the data governance module is mainly designed for further quality control of cooperative medical institutions. In this module, the doctor team or other professionals of the cooperative medical institution will use the visual interface provided by the AI ​​platform to further review, proofread and adjust the data annotation. In addition, this module also provides data preprocessing operations (text preprocessing, image preprocessing) similar to the first data governance module in the technical solution of the aforementioned base model pre-training, which will not be repeated here.

[0027] The downstream task fine-tuning module is used to inherit the model parameters output by the base model pre-training module, and fine-tune the model to obtain an inference model; the design concept of the downstream task fine-tuning module is to allow a team of doctors who lack deep learning experience to train a corresponding model using their own data sets. Taking into account limited computing resources, if multiple cooperative medical institutions have the need to fine-tune the model at the same time, we will have a dedicated team to analyze and schedule experiments based on the urgency of the needs of different institutions, to avoid multiple experiments at the same time, reducing training efficiency and occupying computing resources. Specifically, during the training phase: In this embodiment, the downstream task fine-tuning module will first inherit the model parameters output by the base model pre-training module; then, the model is fine-tuned using the data passed through the second data governance module; instead of fine-tuning all parameters, we only train the decoder: a text decoder based on LLaMA3, an image decoder based on UNETR, and an image decoder based on the MLP classification head; finally, after completing the model training, the base model weights of the base model pre-training module are merged with the fine-tuning model weights of the downstream task fine-tuning module to obtain an inference model.

[0028] The prediction result output module is used to test the performance of the inference model and display the prediction results.

[0029] The prediction result output module is used to test model performance or display prediction results. Specifically, in the inference phase, by adjusting the model state to eval mode, the model will implement the inference task by connecting to different decoders. This module is mainly divided into two steps: The module makes predictions for the input test set data and then outputs the prediction results of the case report, the segmentation results of the lesions and anatomy of the medical image, and the positive and negative classification results; The AI ​​platform will use 3D visualization of nii images to show the difference between the gold standard and the predicted results, and will display the actual results and predicted results of the case report next to the image. If there is no real result data, only the predicted results will be displayed on the platform for users to refer to the output results of the model.

[0030] As another embodiment of the present invention, a method for applying an AI platform for multimodal information fusion includes the following steps: Allowing medical institutions to access and input data, the data including text data and image data, the text data being a diagnosis report, and the image data being 3D medical imaging data; Review and pre-process input data; Inherit the model parameters output by the base model pre-training module in the technical solution of base model pre-training, and fine-tune the model to obtain an inference model; Test the inference model performance and display the prediction results.

[0031] See also Figure 1 As another embodiment of the present invention, an application method of an AI platform for multimodal information fusion includes the following steps: Input data, data preprocessing, the data includes text data and image data, input the text data into a text encoder based on LLaMA3, input the image into a 3D image encoder based on LLaVA-v1.5, construct a similarity matrix, train the model, match the text and image by making the similarity matrix tend to become a unit matrix, fine-tune the text decoder, output the prediction results of the case report, fine-tune the image decoder, output the segmentation results of the lesion and anatomy of the medical image, and the positive and negative classification results.

[0032] As another embodiment of the present invention, in the technical solution implementation of base model training and downstream task fine-tuning, different models can be used to achieve similar effects. For example, in the base model training stage, the LLaMA3 text encoder and the 3D image encoder of LLaVA-v1.5 are respectively changed to text encoders and image encoders based on CLIP and VIT; for example, in the downstream task fine-tuning stage, the UNETR decoder that outputs the segmentation results of lesions or anatomy is replaced by a nnUNet decoder, and the MLP classification head is replaced by another CNN model, etc. The functions are implemented in different ways, but they can output similar results of the present invention, but the effect is not as good as the above-mentioned model.

[0033] In the training and prediction phase of the model, using different training methods for the model only adjusts the direction and method of model convergence, which can also be regarded as alternative solutions. For example, the present invention uses the warm up strategy with the cosine annealing strategy to adjust the learning rate, while the alternative solution can use the natural exponential decay strategy to adjust the learning rate, etc.

[0034] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0035] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An AI platform for multimodal information fusion, characterized in that: The platform includes: A second data input module, used to allow a medical institution to access and input data, wherein the data includes text data and image data, wherein the text data is a diagnosis report, and the image data is 3D medical imaging data; A second data governance module, used for reviewing and preprocessing the data input by the second data input module; The downstream task fine-tuning module is used to inherit the model parameters output by the base model pre-training module in the technical solution of base model pre-training, and fine-tune the model to obtain an inference model; The prediction result output module is used to test the performance of the inference model and display the prediction results.

2. The AI ​​platform for multimodal information fusion according to claim 1, characterized in that: The technical solution for pre-training the base model includes: The first data input module is used to receive data provided by the scientific research platform; The first data governance module is used to control data quality and pre-process data; The base model pre-training module will use the preprocessed data to train a base model and its corresponding parameters for 3D medical imaging data to connect to different downstream tasks.

3. The AI ​​platform for multimodal information fusion according to claim 2, characterized in that: The scientific research platform described in the first data input module is an image data scientific research platform.

4. The AI ​​platform for multimodal information fusion according to claim 2, characterized in that: The first data governance module includes: a data quality control unit that uses human-machine combination to perform efficient and accurate data processing, a standard text data processing unit, and an image data processing unit; Among them, the data quality control unit is controlled by combining human and machine for high efficiency and accuracy: low-quality data is screened through manual annotation process, annotated data with gold standard is obtained, and the anatomical or pathological structure is assisted by using automatic segmentation tools based on SAM; Standard text data processing unit: For the text data of the diagnosis report, the standard token vectorization steps include cleaning useless characters, text segmentation, removing stop words, and vectorization; Image data processing unit: For 3D medical imaging data, the accuracy of the model is improved and I / O conflicts are reduced through data screening, packaging of large files, offline data preprocessing and enhancement. For data screening rules, corresponding screening rules are specified for each body part in the form of a whitelist to ensure that the data that passes the screening is basically directly available data. For offline data preprocessing, the 3D medical imaging data stored in nii format is first loaded, the data is clipped and resized, and then converted into data in numpy array format. The array data of the specified number n is spliced ​​into an array of shape (n, 1, 256, 256, 32) or (n, 1, 256, 256, 128), and finally saved as large files of 5G stored in h5 format.

5. The AI ​​platform for multimodal information fusion according to claim 2, characterized in that: The base model pre-training module constructs a similarity matrix of text-image feature vector pairs by inputting the data processed by the first data management module into the text encoder and the image encoder in sequence; wherein, N text data are output token text vectors through a text encoder that inherits the LLaMA3 parameters, and N 3D image data are output image vectors through a 3D image encoder based on LLaVA-v1.5; the token text vectors and image vectors are used to construct a similarity matrix, wherein each element of the similarity matrix represents the inner product of the corresponding image and text vectors, and during the training process, the inner products on the diagonal represent matching image-text pairs, while the inner products at other positions represent unmatched image-text pairs.

6. The AI ​​platform for multimodal information fusion according to claim 5, characterized in that: In the base model pre-training module, for the modeling of image-text matching, the InfoNCE loss function is set to make the learning goal of the model clear: maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs; in order to prevent the model from overfitting during training, the SiLU activation function, Dropout layer and L2 regularization technology are introduced; In terms of training strategy, we first adopt a phased training method, first inheriting the pre-trained LLaMA3 text encoder, then separately training the 3D image encoder based on LLaVA-v1.5, and finally performing joint image and text training; In addition, mixed precision training is used to speed up the training process while ensuring the accuracy of the model; Use the warm up strategy to gradually increase the learning rate to avoid the initial excessive learning rate affecting model convergence; and use the cosine annealing strategy to dynamically adjust the learning rate to better control the learning rate changes during training and promote efficient convergence of the model; Finally, the training ends and the convergence is judged: the model will stop training when the pre-specified number of epochs is reached; in addition, in order to save computing resources, an early stop strategy is introduced; the model performance is monitored in real time on the validation set, and the training is terminated early when the loss difference between two adjacent epochs is less than 0.0001, thereby avoiding unnecessary training time and possible overfitting.

7. The AI ​​platform for multimodal information fusion according to claim 6, characterized in that: The downstream task fine-tuning module will first inherit the model parameters output by the base model pre-training module; then, the model will be fine-tuned using the data passed through the second data governance module; instead of fine-tuning all parameters, only the decoder will be trained: a text decoder based on LLaMA3, an image decoder based on UNETR, and an image decoder based on an MLP classification head; finally, after completing the model training, the base model weights of the base model pre-training module are merged with the fine-tuning model weights of the downstream task fine-tuning module to obtain an inference model.

8. The AI ​​platform for multimodal information fusion according to claim 1, characterized in that: The prediction result output module is used to make predictions for the input test set data, and then output the prediction results of the case report, the segmentation results of the lesions and anatomy of the medical image, and the positive and negative classification results; The AI ​​platform will use 3D visualization of nii images to show the difference between the gold standard and the predicted results, and will display the actual results and predicted results of the case report next to the image. If there is no real result data, only the predicted results will be displayed on the platform for users to refer to the output results of the model.

9. An application method of an AI platform for multimodal information fusion according to any one of claims 1 to 8, characterized in that: The method comprises the following steps: Allowing medical institutions to access and input data, the data including text data and image data, the text data being a diagnosis report, and the image data being 3D medical imaging data; Review and pre-process input data; Inherit the model parameters output by the base model pre-training module in the technical solution of base model pre-training, and fine-tune the model to obtain an inference model; Test the inference model performance and display the prediction results.

Images