Application updating method of medical image multi-modal large model based on incremental learning

Abstract

The invention relates to model application updating, in particular to an incremental learning-based application updating method for a medical image multi-modal large model, which comprises the following steps of: constructing a multi-modal multi-task data set, and training a medical image multi-modal basic large model by using the multi-modal multi-task data set to obtain a pre-trained medical image multi-modal basic large model; constructing a multi-modal fine tuning data set, and performing fine tuning on the pre-trained medical image multi-modal basic large model by using the multi-modal fine tuning data set to obtain a medical image multi-modal SFT large model; the engine medical image multi-mode SFT large model serves as a medical image multi-mode large model assistant to be deployed on line; according to the technical scheme provided by the invention, the defects in the prior art that catastrophic forgetting of the model on original data is caused after the application is updated, and the performance of the model on typical data of each scene after the application is updated is difficult to ensure can be effectively overcome.

Description

Technical Field

[0001] The present invention relates to model application updating, and in particular to an application updating method of a large multimodal medical image model based on incremental learning. Background Art

[0002] With the continuous advancement of medical imaging technology and its widespread application, a large amount of imaging data, videos and texts have been accumulated in the medical field. However, the number of medical imaging experts has lagged behind, making it difficult to meet the growing demand for medical imaging analysis and reduce the workload of doctors. Therefore, multimodal large medical imaging models for analyzing medical images have gradually been developed and applied.

[0003] The existing iterative update methods for large multimodal models of medical imaging mainly include fine-tuning of all parameters of large models, efficient fine-tuning of parameters of large models, fine-tuning of some modules, updating of model structure or parameter quantity, updating of tasks and supervision methods, etc. The existing data sets used for updating large multimodal models of medical imaging mainly include full data + incremental data, atypical subset data + incremental data, high-quality data + incremental data, incremental data, etc.

[0004] Among the above-mentioned existing iterative update methods of large multimodal medical imaging models, fine-tuning of all parameters of large models has a high performance ceiling, but requires higher economic and time costs; efficient fine-tuning of parameters of large models and fine-tuning of some modules have low costs, but it is difficult to balance the performance of old data and new data; updating of model structure or parameter quantity, tasks and supervision methods requires a lot of experimental verification.

[0005] In addition, among the data sets used for updating existing large multimodal models of medical imaging, full data + incremental data can enable the updated model to obtain comprehensive capabilities, but require more training and updating costs; atypical subset data + incremental data, high-quality data + incremental data can save training and updating costs, but cannot ensure the performance of the model on typical data in each scenario; incremental data can enable the model to obtain better incremental data capabilities, but often leads to catastrophic forgetting of the model on the original data. Summary of the invention

[0006] 1. Technical issues to be solved

[0007] In view of the above-mentioned shortcomings of the prior art, the present invention provides an application update method for a large multimodal model of medical images based on incremental learning, which can effectively overcome the defects of the prior art that the application update causes catastrophic forgetting of the model on the original data, and it is difficult to ensure the performance of the model after the application update on typical data of various scenarios.

[0008] (II) Technical solution

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0010] The application update method of the medical image multimodal large model based on incremental learning includes the following steps:

[0011] S1. Construct a multimodal multitask dataset, and use the multimodal multitask dataset to train a medical imaging multimodal basic large model to obtain a pre-trained medical imaging multimodal basic large model;

[0012] S2. Construct a multimodal fine-tuning dataset, and use the multimodal fine-tuning dataset to fine-tune the pre-trained medical imaging multimodal basic large model to obtain a medical imaging multimodal SFT large model;

[0013] S3. Deploy the engine-based medical image multimodal SFT big model online as a medical image multimodal big model assistant;

[0014] S4. Use the typical data of each scene screened by the online medical imaging multimodal large model assistant, as well as the manually annotated typical data of each scene to build a typical data set;

[0015] S5. Construct an incremental fine-tuning dataset, and incrementally fine-tune the medical imaging multimodal SFT large model through the incremental perceptron based on the incremental fine-tuning dataset, so as to obtain a medical imaging multimodal SFT optimization large model that retains the original data knowledge and learns the incremental data knowledge, and has good performance for typical data of each scene;

[0016] S6. Synchronously update the online medical imaging multimodal big model assistant based on the medical imaging multimodal SFT optimization big model.

[0017] Preferably, in S1, a multimodal multitask dataset is constructed, and the multimodal basic large model of medical images is trained using the multimodal multitask dataset to obtain a pre-trained multimodal basic large model of medical images, including:

[0018] S11. Build a multi-modal and multi-task dataset;

[0019] S12. Use a multimodal multitask dataset to train a visual encoder, a modality adapter of vision and language, and a large language model to obtain a large multimodal basic model of medical imaging with multitask capabilities of medical imaging;

[0020] Among them, the multimodal basic large model of medical imaging includes visual encoder, modality adapter and large language model.

[0021] Preferably, constructing a multimodal multitask dataset in S11 includes:

[0022] Collect 2D medical images, 3D medical images, videos and texts corresponding to each imaging modality, each part examination item and each disease type to construct multimodal data;

[0023] Construct global and local image and text comparison data;

[0024] Construct multi-task data including image interpretation, disease recognition, lesion localization, diagnostic report generation, visual question answering, grounding and referring;

[0025] Construct typical data judgment task data corresponding to each imaging modality, each part examination item and each disease type.

[0026] Preferably, in S2, a multimodal fine-tuning dataset is constructed, and the pre-trained medical image multimodal basic large model is fine-tuned using the multimodal fine-tuning dataset to obtain a medical image multimodal SFT large model, including:

[0027] S21, build a multimodal fine-tuning dataset;

[0028] S22. Use the multimodal fine-tuning dataset to fine-tune one or more of the visual encoder, modality adapter, and large language model to obtain a multimodal SFT large model for medical imaging with higher performance and better specific task capabilities;

[0029] Among them, the fine-tuning methods include full parameter fine-tuning, LoRA parameter efficient fine-tuning, context instruction fine-tuning and RLHF.

[0030] Preferably, constructing a multimodal fine-tuning dataset in S21 includes:

[0031] Collect high-quality multimodal data and task-specific data to build a multimodal fine-tuning dataset.

[0032] Preferably, the engine-based medical image multimodal SFT big model is deployed online as a medical image multimodal big model assistant in S3, including:

[0033] The engine-based medical imaging multimodal SFT big model is used as a medical imaging multimodal big model assistant and deployed to the online medical imaging cloud platform and local equipment of medical institutions to assist doctors in image interpretation, disease identification, lesion localization, diagnosis report generation and explainable report generation.

[0034] Preferably, in S4, typical data of each scene obtained by screening using an online medical imaging multimodal large model assistant and typical data of each scene manually annotated are used to construct a typical data set, including:

[0035] The medical imaging multimodal large model assistant, which has been trained with typical data judgment task data corresponding to each imaging modality, each part examination item and each disease type, is used to filter out typical data for each scene from massive multimodal data;

[0036] At the same time, when using the medical imaging multimodal large model assistant, doctors manually annotate the typical data of each scene, and combine it with the typical data of each scene screened from massive multimodal data to construct a typical data set.

[0037] Preferably, in S5, an incremental fine-tuning dataset is constructed, and the medical image multimodal SFT large model is incrementally fine-tuned through an incremental perceptron based on the incremental fine-tuning dataset, so as to obtain a medical image multimodal SFT optimization large model that retains the original data knowledge and learns the incremental data knowledge, and has good performance for typical data of each scene, including:

[0038] S51, build incremental fine-tuning dataset;

[0039] S52. Adopting the experience replay and parameter isolation methods in incremental learning, freezing the parameters of the medical image multimodal SFT large model, and using the incremental fine-tuning data set to incrementally fine-tune one or more of the visual increment, modality increment, and language increment, to obtain a medical image multimodal SFT optimization large model that retains the original data knowledge and learns the incremental data knowledge, and has good performance for typical data of each scene;

[0040] Among them, the incremental perceptron is a learnable incremental perceptron that controls the activation state of neurons in the network layer, including visual increment, modal increment and language increment. The parameters of the incremental perceptron are fp16 floating-point numbers in the range of [0,1], or 1-bit binary integers with values ​​of 0 or 1, and the initial value is 1.

[0041] Preferably, constructing an incremental fine-tuning dataset in S51 includes:

[0042] An incremental fine-tuning dataset is constructed based on typical datasets and the incremental data returned by the medical imaging multimodal large model assistant.

[0043] Preferably, in S52, the experience replay and parameter isolation methods in incremental learning are used to freeze the parameters of the large multimodal SFT model of medical images, and the incremental fine-tuning data set is used to perform incremental fine-tuning on one or more of the visual increment, the modality increment and the language increment, including:

[0044] The input-output mapping of incremental fine-tuning is expressed as:

[0045] y = Δw·w·x;

[0046] Among them, x is the input of incremental fine-tuning, w represents the large multimodal SFT model of medical images, Δw represents the incremental perceptron, and y is the output of incremental fine-tuning. When the initial value of the incremental perceptron Δw is 1, y=w·x, that is, the initial state of incremental fine-tuning is the large multimodal SFT model of medical images without incremental perceptron.

[0047] (III) Beneficial effects

[0048] Compared with the prior art, the application update method of the medical imaging multimodal big model based on incremental learning provided by the present invention makes full use of the advantages of massive medical imaging multimodal data, incremental medical imaging multimodal data, and medical imaging multimodal big models to obtain a medical imaging multimodal big model assistant that retains massive original medical imaging multimodal data knowledge and can timely and efficiently update incremental data knowledge, so that the updated medical imaging multimodal big model assistant can better retain the original data knowledge and can have better performance on typical data of various scenarios. The application update method provided by the present invention can be applied to the fields of medical imaging analysis and incremental learning, and can also be applied to the research and development and update of medical imaging multimodal big models, which is conducive to the application update of the medical imaging multimodal big model assistant to meet the growing demand for the use of the medical imaging multimodal big model assistant. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Application update method of multimodal large medical imaging model based on incremental learning, such as Figure 1As shown, S1, construct a multimodal multi-task dataset, use the multimodal multi-task dataset to train the medical image multimodal basic large model, and obtain a pre-trained medical image multimodal basic large model, specifically including:

[0053] S11. Build a multi-modal and multi-task dataset;

[0054] S12. Use multimodal and multitask datasets to train visual encoders (such as ViT, CLIP, DINO, etc.), modality adapters of vision and language (such as MLP, Qformer, etc.) and large language models (such as BLOOM, LLaMA, GPT, etc.) to obtain a large multimodal basic model of medical imaging with multitask capabilities.

[0055] Among them, the multimodal basic large model of medical imaging includes visual encoder, modality adapter and large language model.

[0056] Specifically, a multimodal multi-task dataset is constructed in S11, including:

[0057] Collect 2D medical images, 3D medical images, videos and texts corresponding to various imaging modalities (such as X-ray, CT, MRI, ultrasound, pathology, PET, endoscopy, digital subtraction angiography, dermatoscope, etc.), examination items of various parts (such as shoulder joint, elbow joint, lumbar spine, knee joint, ankle joint, cervical spine, thoracic spine, abdomen, pelvis, foot, fibula, hand, nasopharynx, sacrum, radius, wrist joint, humerus, femur, chest, brain, etc.) and various diseases to build multimodal data;

[0058] Construct global and local image and text comparison data;

[0059] Construct multi-task data including image interpretation, disease recognition, lesion localization, diagnostic report generation, visual question answering, grounding and referring;

[0060] Construct typical data judgment task data corresponding to each imaging modality, each part examination item and each disease type.

[0061] S2. Construct a multimodal fine-tuning dataset, and use the multimodal fine-tuning dataset to fine-tune the pre-trained medical imaging multimodal basic large model to obtain a medical imaging multimodal SFT large model, specifically including:

[0062] S21, build a multimodal fine-tuning dataset;

[0063] S22. Use a multimodal fine-tuning dataset to fine-tune one or more of the visual encoder, modality adapter, and large language model to obtain a multimodal SFT large model for medical imaging with higher performance (such as higher accuracy, higher security, higher alignment with human intent, etc.) and better task-specific capabilities;

[0064] Among them, the fine-tuning methods include full parameter fine-tuning, LoRA parameter efficient fine-tuning, context instruction fine-tuning and RLHF.

[0065] Specifically, a multimodal fine-tuning dataset is constructed in S21, including:

[0066] Collect high-quality multimodal data and task-specific data to build a multimodal fine-tuning dataset.

[0067] S3. Deploy the engine-based medical image multimodal SFT big model online as a medical image multimodal big model assistant, including:

[0068] The engine-based medical imaging multimodal SFT big model is used as a medical imaging multimodal big model assistant and deployed to the online medical imaging cloud platform and local equipment of medical institutions to assist doctors in image interpretation, disease identification, lesion localization, diagnosis report generation and explainable report generation.

[0069] S4. Use the typical data of each scene screened by the online medical imaging multimodal large model assistant, as well as the typical data of each scene manually annotated to build a typical data set, including:

[0070] Using the medical imaging multimodal large model assistant trained with typical data judgment task data corresponding to each imaging modality, each part examination item and each disease type, typical data of each scene (such as difficult and complicated disease case data, representative data of each disease type, etc.) is screened from the massive multimodal data;

[0071] At the same time, when using the medical imaging multimodal large model assistant, doctors manually annotate the typical data of each scene, and combine it with the typical data of each scene screened from massive multimodal data to construct a typical data set.

[0072] S5. Construct an incremental fine-tuning dataset, and incrementally fine-tune the multimodal SFT large model of medical images through the incremental perceptron based on the incremental fine-tuning dataset, so as to obtain a multimodal SFT optimization large model of medical images that retains the original data knowledge and learns the incremental data knowledge, and has good performance for typical data of each scene, including:

[0073] S51, build incremental fine-tuning dataset;

[0074] S52. Adopting the experience replay and parameter isolation methods in incremental learning, freezing the parameters of the medical image multimodal SFT large model, and using the incremental fine-tuning data set to incrementally fine-tune one or more of the visual increment, modality increment, and language increment, to obtain a medical image multimodal SFT optimization large model that retains the original data knowledge and learns the incremental data knowledge, and has good performance for typical data of each scene;

[0075] Among them, the incremental perceptron is a learnable incremental perceptron that controls the activation state of neurons in the network layer, including visual increment, modal increment and language increment. The parameters of the incremental perceptron are fp16 floating-point numbers in the range of [0,1], or 1-bit binary integers with values ​​of 0 or 1, and the initial value is 1.

[0076] 1) In S51, an incremental fine-tuning dataset is constructed, including:

[0077] An incremental fine-tuning dataset is constructed based on typical datasets and the incremental data returned by the medical imaging multimodal large model assistant.

[0078] 2) In S52, the experience replay and parameter isolation methods in incremental learning are used to freeze the parameters of the multimodal SFT large model of medical images, and the incremental fine-tuning dataset is used to incrementally fine-tune one or more of the visual increment, modality increment, and language increment, including:

[0079] The input-output mapping of incremental fine-tuning is expressed as:

[0080] y = Δw·w·x;

[0081] Among them, x is the input of incremental fine-tuning, w represents the large multimodal SFT model of medical images, Δw represents the incremental perceptron, and y is the output of incremental fine-tuning. When the initial value of the incremental perceptron Δw is 1, y=w·x, that is, the initial state of incremental fine-tuning is the large multimodal SFT model of medical images without incremental perceptron.

[0082] S6. Synchronously update the online medical imaging multimodal big model assistant based on the medical imaging multimodal SFT optimization big model.

[0083] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An application update method for a multimodal large model of medical images based on incremental learning, characterized by: The following steps are involved: S1. Construct a multimodal multitask dataset, and use the multimodal multitask dataset to train a medical imaging multimodal basic large model to obtain a pre-trained medical imaging multimodal basic large model; S2. Construct a multimodal fine-tuning dataset, and use the multimodal fine-tuning dataset to fine-tune the pre-trained medical imaging multimodal basic large model to obtain a medical imaging multimodal SFT large model; S3. Deploy the engine-based medical image multimodal SFT big model online as a medical image multimodal big model assistant; S4. Use the typical data of each scene screened by the online medical imaging multimodal large model assistant, as well as the manually annotated typical data of each scene to build a typical data set; S5. Construct an incremental fine-tuning dataset, and incrementally fine-tune the medical imaging multimodal SFT large model through the incremental perceptron based on the incremental fine-tuning dataset, so as to obtain a medical imaging multimodal SFT optimization large model that retains the original data knowledge and learns the incremental data knowledge, and has good performance for typical data of each scene; S6. Synchronously update the online medical imaging multimodal big model assistant based on the medical imaging multimodal SFT optimization big model.

2. The application update method of the medical image multimodal large model based on incremental learning according to claim 1 is characterized by: In S1, a multimodal multitask dataset is constructed, and the multimodal basic large model of medical imaging is trained using the multimodal multitask dataset to obtain a pre-trained multimodal basic large model of medical imaging, including: S11. Build a multi-modal and multi-task dataset; S12. Use a multimodal multitask dataset to train a visual encoder, a modality adapter of vision and language, and a large language model to obtain a large multimodal basic model of medical imaging with multitask capabilities of medical imaging; Among them, the multimodal basic large model of medical imaging includes visual encoder, modality adapter and large language model.

3. The application update method of the medical image multimodal large model based on incremental learning according to claim 2 is characterized by: In S11, a multi-modal and multi-task dataset is constructed, including: Collect 2D medical images, 3D medical images, videos and texts corresponding to each imaging modality, each part examination item and each disease type to construct multimodal data; Construct global and local image and text comparison data; Construct multi-task data including image interpretation, disease recognition, lesion localization, diagnostic report generation, visual question answering, grounding and referring; Construct typical data judgment task data corresponding to each imaging modality, each part examination item and each disease type.

4. The application update method of the medical image multimodal large model based on incremental learning according to claim 2 is characterized by: In S2, a multimodal fine-tuning dataset is constructed, and the pre-trained medical imaging multimodal basic large model is fine-tuned using the multimodal fine-tuning dataset to obtain a medical imaging multimodal SFT large model, including: S21, build a multimodal fine-tuning dataset; S22. Use the multimodal fine-tuning dataset to fine-tune one or more of the visual encoder, modality adapter, and large language model to obtain a multimodal SFT large model for medical imaging with higher performance and better specific task capabilities; Among them, the fine-tuning methods include full parameter fine-tuning, LoRA parameter efficient fine-tuning, context instruction fine-tuning and RLHF.

5. The application update method of the medical image multimodal large model based on incremental learning according to claim 4 is characterized by: In S21, a multimodal fine-tuning dataset is constructed, including: Collect high-quality multimodal data and task-specific data to build a multimodal fine-tuning dataset.

6. The application update method of the medical image multimodal large model based on incremental learning according to claim 4 is characterized by: In S3, the engine-based medical imaging multimodal SFT big model is deployed online as a medical imaging multimodal big model assistant, including: The engine-based medical imaging multimodal SFT big model is used as a medical imaging multimodal big model assistant and deployed to the online medical imaging cloud platform and local equipment of medical institutions to assist doctors in image interpretation, disease identification, lesion localization, diagnosis report generation and explainable report generation.

7. The application update method of the medical image multimodal large model based on incremental learning according to claim 6 is characterized by: In S4, typical data of each scene are obtained by screening using the online medical imaging multimodal large model assistant, as well as manually labeled typical data of each scene, to construct a typical data set, including: The medical imaging multimodal large model assistant, which has been trained with typical data judgment task data corresponding to each imaging modality, each part examination item and each disease type, is used to filter out typical data for each scene from massive multimodal data; At the same time, when using the medical imaging multimodal large model assistant, doctors manually annotate the typical data of each scene, and combine it with the typical data of each scene screened from massive multimodal data to construct a typical data set.

8. The application update method of the medical image multimodal large model based on incremental learning according to claim 7 is characterized by: In S5, an incremental fine-tuning dataset is constructed. Based on the incremental fine-tuning dataset, the multimodal SFT large model of medical images is incrementally fine-tuned through the incremental perceptron. The multimodal SFT optimization large model of medical images is obtained, which retains the original data knowledge and learns the incremental data knowledge. At the same time, it has good performance for typical data of each scene, including: S51, build incremental fine-tuning dataset; S52. Adopting the experience replay and parameter isolation methods in incremental learning, freezing the parameters of the medical image multimodal SFT large model, and using the incremental fine-tuning data set to incrementally fine-tune one or more of the visual increment, modality increment, and language increment, to obtain a medical image multimodal SFT optimization large model that retains the original data knowledge and learns the incremental data knowledge, and has good performance for typical data of each scene; Among them, the incremental perceptron is a learnable incremental perceptron that controls the activation state of neurons in the network layer, including visual increment, modal increment and language increment. The parameters of the incremental perceptron are fp16 floating-point numbers in the range of [0,1], or 1-bit binary integers with values ​​of 0 or 1, and the initial value is 1.

9. The application update method of the medical image multimodal large model based on incremental learning according to claim 8 is characterized by: In S51, we construct an incremental fine-tuning dataset, including: An incremental fine-tuning dataset is constructed based on typical datasets and the incremental data returned by the medical imaging multimodal large model assistant.

10. The application update method of the medical image multimodal large model based on incremental learning according to claim 8, characterized in that: S52 uses the experience replay and parameter isolation methods in incremental learning to freeze the parameters of the multimodal SFT large model of medical images, and uses the incremental fine-tuning dataset to incrementally fine-tune one or more of the visual increment, modality increment, and language increment, including: The input-output mapping of incremental fine-tuning is expressed as: y = Δw·w·x; Among them, x is the input of incremental fine-tuning, w represents the large multimodal SFT model of medical images, Δw represents the incremental perceptron, and y is the output of incremental fine-tuning. When the initial value of the incremental perceptron Δw is 1, y=w·x, that is, the initial state of incremental fine-tuning is the large multimodal SFT model of medical images without incremental perceptron.

Images