Application updating method of medical image multi-modal large model based on incremental learning

Abstract

The application relates to model application updating, in particular to an application updating method of a medical image multi-modal large model based on incremental learning, constructing a multi-modal multi-task data set, training a medical image multi-modal basic large model by using the multi-modal multi-task data set, obtaining a pre-trained medical image multi-modal basic large model, constructing a multi-modal fine-tuning data set, fine-tuning the pre-trained medical image multi-modal basic large model by using the multi-modal fine-tuning data set, obtaining a medical image multi-modal SFT large model, and deploying the engineized medical image multi-modal SFT large model to online as a medical image multi-modal large model assistant. The technical scheme provided by the application can effectively overcome the defects that the existing technology causes catastrophic forgetting of a model on original data after application updating and is difficult to ensure the performance of the model on typical data in various scenes after application updating.

Description

TECHNICAL FIELD

[0001] The present application relates to model application updating, in particular to an application updating method of a medical image multi-modal large model based on incremental learning. BACKGROUND

[0002] With the continuous progress and wide application of medical imaging technology, a large amount of image data, video and text has been accumulated in the medical field. However, the number of medical image experts grows relatively slowly, which is difficult to meet the increasing demand for medical image analysis and reduce the workload of doctors. Therefore, medical image multi-modal large models for analyzing medical images have been gradually developed and applied.

[0003] The existing iterative updating methods of medical image multi-modal large models mainly include large model full parameter fine-tuning, large model parameter efficient fine-tuning, partial module fine-tuning, model structure or parameter quantity updating, task and supervision method updating, etc. The existing data sets used for updating medical image multi-modal large models mainly include full data + incremental data, non-typical subset data + incremental data, high-quality data + incremental data, and incremental data.

[0004] Among the above-mentioned existing iterative updating methods of medical image multi-modal large models, the performance upper limit of large model full parameter fine-tuning is high, but it consumes higher economic and time costs; the cost of large model parameter efficient fine-tuning and partial module fine-tuning is lower, but it is difficult to balance the performance of old data and new data; model structure or parameter quantity updating and task and supervision method updating require a large number of experimental verifications.

[0005] In addition, in the existing data sets used for updating medical image multi-modal large models, full data + incremental data can enable the updated model to obtain comprehensive capabilities, but it requires more training and updating costs; non-typical subset data + incremental data and high-quality data + incremental data can save training and updating costs, but they cannot ensure the performance of the model on typical data in each scene; incremental data can enable the model to obtain better incremental data capabilities, but it often leads to catastrophic forgetting of the model on the original data. SUMMARY

[0006] (I) Technical problems to be solved

[0007] In view of the above-mentioned shortcomings of the prior art, the present application provides an application updating method of a medical image multi-modal large model based on incremental learning, which can effectively overcome the defects of the prior art that the application updating leads to catastrophic forgetting of the model on the original data and the difficulty to ensure the performance of the model on typical data in each scene after application updating.

[0008] (II) Technical solutions

[0009] To achieve the above object, the present application is realized by the following technical solutions:

[0010] The application updating method of the medical image multi-modal large model based on incremental learning comprises the following steps:

[0011] S1, a multi-modal multi-task data set is constructed, a medical image multi-modal basic large model is trained using the multi-modal multi-task data set, and a pre-trained medical image multi-modal basic large model is obtained;

[0012] S2, a multi-modal fine-tuning data set is constructed, the pre-trained medical image multi-modal basic large model is fine-tuned using the multi-modal fine-tuning data set, and a medical image multi-modal SFT large model is obtained;

[0013] S3, the engineized medical image multi-modal SFT large model is deployed as a medical image multi-modal large model assistant online;

[0014] S4, the typical data set is constructed using the medical image multi-modal large model assistant online and the artificial labeled typical data of each scene;

[0015] S5, an incremental fine-tuning data set is constructed, the medical image multi-modal SFT large model is incrementally fine-tuned based on the incremental fine-tuning data set through an incremental perceiver, and a medical image multi-modal SFT optimization large model is obtained, which retains the original data knowledge and learns the incremental data knowledge, and has good performance for each scene typical data;

[0016] S6, the medical image multi-modal SFT optimization large model is used to synchronize the update of the medical image multi-modal large model assistant online.

[0017] Preferably, in S1, the multi-modal multi-task data set is constructed, the medical image multi-modal basic large model is trained using the multi-modal multi-task data set, and the pre-trained medical image multi-modal basic large model is obtained, comprising:

[0018] S11, a multi-modal multi-task data set is constructed;

[0019] S12, the visual encoder, the visual and language modal adapter and the large language model are trained using the multi-modal multi-task data set, and the medical image multi-modal basic large model with medical image multi-task capability is obtained;

[0020] The medical image multi-modal basic large model comprises a visual encoder, a modal adapter and a large language model.

[0021] Preferably, in S11, the multi-modal multi-task data set is constructed, comprising:

[0022] Collect 2D medical images, 3D medical images, videos and texts corresponding to each imaging modality, each part examination item and each disease to construct multi-modal data;

[0023] Construct global and local text-image contrast data;

[0024] Construct multi-task data including image interpretation, disease identification, lesion positioning, diagnosis report generation, visual question answering, grounding and referring;

[0025] Construct typical data judgment task data corresponding to each imaging modality, each part examination item and each disease.

[0026] Preferably, in S2, a multi-modal fine-tuning data set is constructed, and the pre-trained medical image multi-modal base large model is fine-tuned using the multi-modal fine-tuning data set to obtain a medical image multi-modal SFT large model, comprising:

[0027] S21, constructing a multi-modal fine-tuning data set;

[0028] S22, fine-tuning one or more of the visual encoder, the modality adapter and the large language model using the multi-modal fine-tuning data set to obtain a medical image multi-modal SFT large model with higher performance and better specific task capability;

[0029] Wherein, the fine-tuning mode includes full-parameter fine-tuning, LoRA parameter efficient fine-tuning, context instruction fine-tuning and RLHF.

[0030] Preferably, in S21, the multi-modal fine-tuning data set is constructed, comprising:

[0031] Collect high-quality multi-modal data and specific task data to construct a multi-modal fine-tuning data set.

[0032] Preferably, in S3, the engineized medical image multi-modal SFT large model is deployed to the online as a medical image multi-modal large model assistant, comprising:

[0033] The engineized medical image multi-modal SFT large model is deployed to the online medical image cloud platform and medical institution local equipment as a medical image multi-modal large model assistant to assist doctors in image interpretation, disease identification, lesion positioning, diagnosis report generation and explainable report generation.

[0034] Preferably, in S4, the typical data set is constructed using the typical data of each scene screened by the online medical image multi-modal large model assistant and the artificial labeled typical data of each scene, comprising:

[0035] The medical image multi-modal large model assistant trained by the typical data judgment task data corresponding to each imaging modality, each part examination item and each disease type is used to screen the typical data of each scene from the massive multi-modal data;

[0036] Meanwhile, the doctors manually annotate the typical data of each scene during the use of the medical image multi-modal large model assistant, and combine the typical data of each scene obtained from the massive multi-modal data to construct a typical data set.

[0037] Preferably, an incremental fine-tuning data set is constructed in S5, and the medical image multi-modal SFT large model is incrementally fine-tuned by an incremental perceiver based on the incremental fine-tuning data set, to obtain a medical image multi-modal SFT optimized large model that retains the original data knowledge and learns the incremental data knowledge, and has good performance for the typical data of each scene, including:

[0038] S51, constructing an incremental fine-tuning data set;

[0039] S52, using the experience replay and parameter isolation methods in incremental learning, freezing the parameters of the medical image multi-modal SFT large model, and incrementally fine-tuning one or more of visual increment, modality increment and language increment using the incremental fine-tuning data set, to obtain a medical image multi-modal SFT optimized large model that retains the original data knowledge and learns the incremental data knowledge, and has good performance for the typical data of each scene;

[0040] The incremental perceiver is a learnable incremental perceiver that controls the activation state of the network layer neuron, and includes visual increment, modality increment and language increment. The parameters of the incremental perceiver are fp16 floating point numbers in the range of [0, 1], or 1-bit binary integers with values of 0 and 1, and the initial value is 1.

[0041] Preferably, the incremental fine-tuning data set is constructed in S51, including:

[0042] According to the typical data set and the incremental data returned by the medical image multi-modal large model assistant, an incremental fine-tuning data set is constructed.

[0043] Preferably, in S52, the experience replay and parameter isolation methods in incremental learning are used to freeze the parameters of the medical image multi-modal SFT large model, and one or more of visual increment, modality increment and language increment are incrementally fine-tuned using the incremental fine-tuning data set, including:

[0044] The input-output mapping of incremental fine-tuning is represented as:

[0045] y=Δw·w·x;

[0046] Wherein, x is the input of incremental fine-tuning, w represents the medical image multi-modal SFT large model, Aw represents the incremental perceiver, y is the output of incremental fine-tuning, when the initial value of the incremental perceiver Aw is 1, y=w·x, that is, the initial state of the incremental fine-tuning is the medical image multi-modal SFT large model without the incremental perceiver.

[0047] (III) Beneficial Effects

[0048] Compared with the prior art, the application update method of the medical image multi-modal large model based on incremental learning provided by the application fully utilizes the advantages of massive medical image multi-modal data, incremental medical image multi-modal data, and the medical image multi-modal large model, obtains a medical image multi-modal large model assistant that retains the knowledge of massive original medical image multi-modal data and can update the knowledge of incremental data in a timely and efficient manner, so that the updated medical image multi-modal large model assistant can better retain the original data knowledge and have good performance on typical data in various scenarios. The application update method provided by the application can be applied to the fields of medical image analysis and incremental learning, and can also be applied to the research and update of the medical image multi-modal large model, which is helpful for the application update of the medical image multi-modal large model assistant to meet the increasing use demand of the medical image multi-modal large model assistant. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0050] Figure 1 The flowchart of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0052] The application update method of the medical image multi-modal large model based on incremental learning, such as Figure 1As shown, S1, a multi-modal multi-task data set is constructed, a medical image multi-modal base large model is trained using the multi-modal multi-task data set, and a pre-trained medical image multi-modal base large model is obtained, specifically including:

[0053] S11, a multi-modal multi-task data set is constructed;

[0054] S12, a visual encoder (such as ViT, CLIP, DINO, etc.), a modal adapter (such as MLP, Qformer, etc.) of vision and language, and a large language model (such as BLOOM, LLaMA, GPT, etc.) are trained using the multi-modal multi-task data set, and a medical image multi-modal base large model with medical image multi-task capability is obtained;

[0055] The medical image multi-modal base large model includes a visual encoder, a modal adapter, and a large language model.

[0056] Specifically, the multi-modal multi-task data set is constructed in S11, including:

[0057] 2D medical images, 3D medical images, videos, and texts corresponding to each imaging modality (such as X-ray, CT, MRI, ultrasound, pathology, PET, endoscopy, digital subtraction angiography, dermatoscope, etc.), each part (such as shoulder joint, elbow joint, lumbar spine, knee joint, ankle joint, cervical spine, thoracic spine, abdomen, pelvis, foot, fibula, hand, nasopharynx, sacrococcygeal bone, ulnar radius, wrist joint, humerus, femur, chest, brain, etc.) examination items and each disease are collected to construct multi-modal data;

[0058] Global and local image-text comparison data are constructed;

[0059] Multi-task data including image interpretation, disease recognition, lesion positioning, diagnosis report generation, visual question answering, Grounding, and Referring are constructed;

[0060] Typical data judgment task data corresponding to each imaging modality, each part examination item, and each disease are constructed.

[0061] S2, a multi-modal fine-tuning data set is constructed, and the pre-trained medical image multi-modal base large model is fine-tuned using the multi-modal fine-tuning data set to obtain a medical image multi-modal SFT large model, specifically including:

[0062] S21, a multi-modal fine-tuning data set is constructed;

[0063] S22, one or more of the visual encoder, the modal adapter, and the large language model are fine-tuned using the multi-modal fine-tuning data set to obtain a medical image multi-modal SFT large model with higher performance (such as higher accuracy, higher security, higher alignment with human intent, etc.), better specific task capability.

[0064] The fine-tuning manner includes full-parameter fine-tuning, LoRA parameter efficient fine-tuning, context instruction fine-tuning and RLHF.

[0065] Specifically, the multi-modal fine-tuning dataset is constructed in S21, including:

[0066] High-quality multi-modal data and specific task data are collected to construct the multi-modal fine-tuning dataset.

[0067] S3, the engineized medical image multi-modal SFT large model is deployed as a medical image multi-modal large model assistant to the online, specifically including:

[0068] The engineized medical image multi-modal SFT large model is deployed as a medical image multi-modal large model assistant to the online medical image cloud platform and local medical institution equipment to assist doctors in image interpretation, disease identification, lesion positioning, diagnosis report generation and explainable report generation.

[0069] S4, using the typical data of each scene obtained by screening the online medical image multi-modal large model assistant and the artificial labeled typical data of each scene, a typical dataset is constructed, specifically including:

[0070] Using the typical data corresponding to each imaging modality, each part examination item and each disease to judge the medical image multi-modal large model assistant trained by the task data to screen the typical data of each scene (such as difficult and complex disease case data, representative data of each disease, etc.) from the massive multi-modal data;

[0071] At the same time, doctors manually label the typical data of each scene during the use of the medical image multi-modal large model assistant, and combine it with the typical data of each scene screened from the massive multi-modal data to construct a typical dataset.

[0072] S5, constructing an incremental fine-tuning dataset, based on the incremental fine-tuning dataset, the medical image multi-modal SFT large model is incrementally fine-tuned by an incremental perceiver to obtain a medical image multi-modal SFT optimization large model that retains the original data knowledge and learns the incremental data knowledge and has good performance for each scene typical data, specifically including:

[0073] S51, constructing an incremental fine-tuning dataset;

[0074] S52, using the experience replay and parameter isolation method in incremental learning, freezing the parameters of the medical image multi-modal SFT large model, using the incremental fine-tuning dataset to fine-tune one or more of the visual increment, modal increment and language increment, obtaining a medical image multi-modal SFT optimization large model that retains the original data knowledge and learns the incremental data knowledge, and has good performance for typical data in each scene;

[0075] Wherein, the incremental perceiver is a learnable incremental perceiver for controlling the activation state of the network layer neuron, including visual increment, modal increment and language increment, the parameters of the incremental perceiver are fp16 floating point numbers in the range of [0, 1], or 1-bit binary integers with values of 0 and 1, and the initial value is 1.

[0076] 1) S51 constructs an incremental fine-tuning dataset, including:

[0077] According to the typical dataset and the incremental data returned by the medical image multi-modal large model assistant, the incremental fine-tuning dataset is constructed.

[0078] 2) S52 uses the experience replay and parameter isolation method in incremental learning, freezes the parameters of the medical image multi-modal SFT large model, and uses the incremental fine-tuning dataset to fine-tune one or more of the visual increment, modal increment and language increment, including:

[0079] The input-output mapping of incremental fine-tuning is represented as:

[0080] y = Δw·w·x;

[0081] Wherein, x is the input of incremental fine-tuning, w represents the medical image multi-modal SFT large model, Δw represents the incremental perceiver, and y is the output of incremental fine-tuning. When the initial value of the incremental perceiver Δw is 1, y = w·x, that is, the initial state of incremental fine-tuning is the medical image multi-modal SFT large model without incremental perceiver.

[0082] S6, according to the medical image multi-modal SFT optimization large model, the on-line medical image multi-modal large model assistant is updated synchronously.

[0083] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements will not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An application updating method of a medical image multi-modal large model based on incremental learning, characterized in that: Comprise the following steps: S1, construct a multi-modal multi-task data set, train the medical image multi-modal base large model using the multi-modal multi-task data set, and obtain a pre-trained medical image multi-modal base large model; S2, construct a multi-modal fine-tuning data set, fine-tune the pre-trained medical image multi-modal base large model using the multi-modal fine-tuning data set, and obtain a medical image multi-modal SFT large model, comprising: S21, constructing a multi-modal fine-tuning data set; S22, fine-tune one or more of the visual encoder, the modal adapter, and the large language model using the multi-modal fine-tuning data set to obtain a medical image multi-modal SFT large model with higher performance and better specific task capability; Wherein, the fine-tuning mode includes full-parameter fine-tuning, LoRA parameter efficient fine-tuning, context instruction fine-tuning and RLHF; S3, deploy the engine-based medical image multi-modal SFT large model as a medical image multi-modal large model assistant to the online; S4, using the medical image multi-modal large model assistant on line to screen the typical data of each scene, and the artificial labeled typical data of each scene, construct a typical data set; S5, construct an incremental fine-tuning data set, and based on the incremental fine-tuning data set, perform incremental fine-tuning on the medical image multi-modal SFT large model through an incremental perceiver to obtain a medical image multi-modal SFT optimization large model that retains the original data knowledge and learns the incremental data knowledge, and has good performance for each scene typical data, comprising: S51, constructing an incremental fine-tuning data set: According to the typical data set, and the incremental data returned by the medical image multi-modal large model assistant, an incremental fine-tuning data set is constructed; S52, using the experience replay and parameter isolation methods in incremental learning, freezing the parameters of the medical image multi-modal SFT large model, and using the incremental fine-tuning data set to perform incremental fine-tuning on one or more of the visual increment, the modal increment and the language increment, to obtain a medical image multi-modal SFT optimization large model that retains the original data knowledge and learns the incremental data knowledge, and has good performance for each scene typical data; Wherein, the incremental perceiver is a learnable incremental perceiver that controls the activation state of the network layer neuron, including visual increment, modal increment and language increment, the parameters of the incremental perceiver are fp16 floating point numbers in the range of [0, 1], or 1-bit binary integers with values of 0 and 1, and the initial value is 1; S6, according to the medical image multi-modal SFT optimization large model, synchronously update the medical image multi-modal large model assistant on line.

2. The application updating method of the medical image multi-modal large model based on incremental learning according to claim 1, characterized in that: In S1, the multi-modal multi-task data set is constructed, the medical image multi-modal base large model is trained using the multi-modal multi-task data set, and the pre-trained medical image multi-modal base large model is obtained, comprising: S11, constructing a multi-modal multi-task data set; S12, train the visual encoder, the modal adapter of vision and language, and the large language model using the multi-modal multi-task data set to obtain a medical image multi-modal base large model with medical image multi-task capability; Wherein, the medical image multi-modal base large model comprises a visual encoder, a modal adapter and a large language model.

3. The application updating method of the medical image multi-modal large model based on incremental learning according to claim 2, characterized in that: In S11, a multi-modal multi-task data set is constructed, including: 2D medical images, 3D medical images, videos and texts corresponding to each imaging modality, each part examination item and each disease are collected to construct multi-modal data; Global and local image-text comparison data are constructed; Multi-task data including image interpretation, disease recognition, lesion positioning, diagnosis report generation, visual question answering, grounding and referring are constructed; Typical data judgment task data corresponding to each imaging modality, each part examination item and each disease are constructed. 4.The application updating method of a medical image multi-modal large model based on incremental learning according to claim 1, characterized in that: In S21, a multi-modal fine-tuning data set is constructed, including: High-quality multi-modal data and specific task data are collected to construct a multi-modal fine-tuning data set. 5.The application updating method of a medical image multi-modal large model based on incremental learning according to claim 1, characterized in that: In S3, the engine-based medical image multi-modal SFT large model is deployed as a medical image multi-modal large model assistant online, including: The engine-based medical image multi-modal SFT large model is deployed as a medical image multi-modal large model assistant in the online medical image cloud platform and local medical institution equipment to assist doctors in image interpretation, disease recognition, lesion positioning, diagnosis report generation and interpretable report generation.

6. The application updating method of the medical image multi-modal large model based on incremental learning according to claim 5, characterized in that: In S4, the typical data sets are constructed using the typical data of each scene screened by the online medical image multi-modal large model assistant and the typical data of each scene artificially annotated, including: The medical image multi-modal large model assistant trained by the typical data judgment task data corresponding to each imaging modality, each part examination item and each disease is used to screen the typical data of each scene from the massive multi-modal data; At the same time, doctors manually annotate the typical data of each scene during the use of the medical image multi-modal large model assistant, and combine it with the typical data of each scene screened from the massive multi-modal data to construct a typical data set. 7.The application updating method of a medical image multi-modal large model based on incremental learning according to claim 1, characterized in that: In S52, the experience replay and parameter isolation methods in incremental learning are used to freeze the parameters of the medical image multi-modal SFT large model, and one or more of visual increment, modality increment and language increment are fine-tuned using the incremental fine-tuning data set, including: The input-output mapping of the incremental fine-tuning is represented as: wherein, is the input of the incremental fine-tuning, represents the medical image multi-modal SFT large model, represents the incremental perceiver, is the output of the incremental fine-tuning, when the incremental perceiver is the initial value of 1, that is, the initial state of the incremental fine-tuning is the medical image multi-modal SFT large model without the incremental perceiver.

Images