Medical image continuous learning classification method based on text prototype guidance

Abstract

The invention discloses a medical image continuous learning classification method based on text prototype guidance, and the method comprises the steps: a model training stage: constructing a continuous learning frame which comprises an image description generation module, a text encoder and a visual encoder; in the continuous learning framework, generating a category text prototype corresponding to the image set of the specific category, using the category text prototype as a guide signal of visual encoder learning, and integrating a gradient weighting mechanism to realize adapter fusion; in an inference phase, a task sharing adapter and a frozen VLM text / visual encoder are jointly used to identify test images from any learned t tasks. According to the method, description which is output by a large language model and is irrelevant to disease symptoms or wrong description can be overcome, so that natural description can be coded more effectively.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and specifically relates to a medical image continuous learning and classification method based on text prototype guidance. Background Art

[0002] Deep neural networks have recently demonstrated remarkable performance in disease diagnosis based on medical imaging. However, due to the challenges of collecting medical imaging data covering all disease categories for a single organ or tissue (such as the skin), existing intelligent diagnostic systems are typically only able to identify a limited number of disease categories. To address this issue, continuous learning techniques are being applied to diagnostic systems, enabling them to incrementally learn new disease categories. A core challenge in this area is mitigating catastrophic forgetting, which occurs when a model's recognition performance for old categories degrades dramatically after learning new ones.

[0003] Existing research has proposed three main solutions: Regularization-based methods maintain the original knowledge by penalizing significant changes in key parameters of old categories, but the introduction of regularization terms may inhibit the model's ability to adapt to new knowledge, leading to suboptimal continuous learning performance. Replay-based methods retain a small number of old category samples and implement knowledge replay by jointly training with new data. They are often combined with knowledge distillation techniques to achieve knowledge transfer from old models to new models. Parameter expansion-based methods learn new knowledge by freezing old modules and adding task-specific modules, but they face difficulties in module selection during inference and the problem of model size and latency increasing with the learning stage.

[0004] Inspired by the language-assisted visual cognition mechanism in human intelligence, some studies have used pre-trained language models to encode category names and generate semantic objectives to guide visual representation learning. However, in medical scenarios, most visual-language models (VLMs) often lack professional medical knowledge. Fixed cues such as "a photo of [category name]" may not accurately provide semantic information about the disease, thus hindering the learning process of the visual encoder. Some methods have attempted to use large language models (such as ChatGPT) to generate more diverse category descriptions to avoid the semantic information bias caused by fixed cues. However, these methods are still limited by the fact that large language models often output irrelevant or incorrect descriptions of disease symptoms. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a method for continuous learning and classification of medical images based on text prototypes. When learning a set of new diseases, this method first uses image description models such as BLIP-2 and LLaVa[11,15] to generate descriptive text for each training image. These descriptions can generally summarize the disease symptoms appearing in the image in popular natural language. Subsequently, by summarizing the descriptive text of all images in the same category, a text prototype for each category can be obtained. These textual prototypes are then used in the continuous learning framework to guide the training of the visual encoder in the continuous learning process. This can overcome the irrelevant or erroneous descriptions of the disease symptoms output by the large language model, thereby more effectively encoding these natural descriptions.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for continuous learning and classification of medical images based on text prototype guidance, comprising the following steps:

[0008] Model training phase:

[0009] Build a continuous learning framework, including an image description generation module, a text encoder, and a visual encoder;

[0010] In a continuous learning framework, we generate category text prototypes corresponding to a set of images of a specific category, use the category text prototypes as guidance signals for visual encoder learning, and integrate a gradient weighting mechanism to achieve adapter fusion;

[0011] Reasoning stage:

[0012] Freeze the image description generation module and text encoder, input the test image set into the visual encoder that has completed adapter fusion, obtain the image encoding, align the category text prototype with the image encoding, obtain the first value, pass the image encoding through the fully connected layer to output the second value, normalize the first value and the second value respectively, and obtain the prediction result after calculation.

[0013] As a preferred technical solution, generating a category text prototype corresponding to a specific category of image set includes:

[0014] During the learning task, the image description generation module is used to process the image set of a specific category to obtain the corresponding text description. The category name of the specific category image is spliced ​​with the corresponding text description to obtain the enhanced text description. The text encoder is used to extract the text embedding corresponding to the enhanced text description, and all text embeddings are dedifferentiated to obtain the category text prototype.

[0015] As a preferred technical solution, all text embeddings are dedifferentiated as follows:

[0016] Average all text embeddings of the kth new category to obtain the category text prototype in Represents the set of all image indices belonging to the kth new category.

[0017] As a preferred technical solution, the method of using the category text prototype as a guiding signal for visual encoder learning includes:

[0018] Use CLIP style loss function to guide the visual encoder f v The training is as follows

[0019]

[0020] Among them, z n =f V (x n ) represents the training image X n Image encoding, cos(·,·) is the cosine similarity, N t is the total number of training images in task t, y n is x n The category label, is the indicator function.

[0021] As a preferred technical solution, it also includes inputting the image encoding into the fully connected layer, obtaining logits and calculating the cross entropy loss The total loss of the visual encoder training is Where λ is a constant coefficient that balances the two losses.

[0022] As a preferred technical solution, the visual encoder learning includes multiple adapters, each adapter includes multiple fully connected layers and activation functions; the learnable adapter is set in each layer of the visual encoder.

[0023] As a preferred technical solution, the integrated gradient weighting mechanism realizes adapter fusion, including:

[0024] Importance evaluation: Calculate the gradient of the adapter parameters by setting the loss function to obtain the first gradient matrix, square all elements of the first gradient matrix to obtain the second gradient matrix;

[0025] Adapter fusion: All adapter parameters {W 1，l ,...,W t，l} to merge and obtain the task sharing adapter as follows:

[0026]

[0027] Among them, ⊙ represents the element-by-element product, and both the numerator and denominator are element-by-element operations. s，l represents the gradient matrix of the adapter for the s-th learning task at the l-th layer, A′ j，l The gradient matrix of the adapter for the jth learning task at layer l, W l Will be used as parameters for the shared adapter of layer l tasks.

[0028] As a preferred technical solution, the gradient of the adapter parameter is calculated as follows:

[0029]

[0030] in, represents the loss function, W t，l Represents adapter parameters.

[0031] As a preferred technical solution, it also includes: sampling a balanced subset of categories from new and old data, and minimizing the loss Fine-tune the task sharing adapter.

[0032] As a preferred technical solution, the first value and the second value are normalized separately and the prediction result is obtained after calculation, specifically:

[0033] Perform softmax operations on the first and second values ​​respectively, sum the obtained probability vectors, and take the category corresponding to the maximum value as the prediction result.

[0034] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0035] (1) The continuous learning framework (TPGCL) proposed in this paper uses text prototypes containing rich semantic information about disease symptoms to guide the training of visual encoders. By calculating the mean of the text embeddings of all image annotations belonging to the same disease category, text prototypes with discriminative semantic information can be obtained. At the same time, based on the training of text prototypes, compared with the descriptions generated by fixed prompt words or large language models, the present invention can accurately describe the typical features of the image in natural language.

[0036] (2) The present invention proposes a gradient weighting mechanism, which can ensure that parameters important to the old knowledge are fully preserved while merging multiple old adapters into new adapters, and effectively alleviate the problem of catastrophic forgetting. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 This is a flowchart of a method for continuous learning and classification of medical images guided by text prototypes according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0040] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0041] See also Figure 1 This embodiment provides a continuous learning and classification method for medical images guided by text prototypes. The method is divided into two phases: the first is the model training phase, and the second is the inference phase. In the model training phase, text prototypes are used as auxiliary supervisory signals to train the visual encoder, and multiple old adapters are merged into new adapters through a gradient weighted fusion mechanism. In the inference phase, the task-sharing adapter is mainly used together with the frozen VLM text / visual encoder to recognize test images from any of the t learned tasks.

[0042] Specifically, during the model training phase, this embodiment constructs a continuous learning framework, TPGCL, which includes an image description generation module, a text encoder, and a visual encoder. This framework uses text prototypes as guiding signals for visual encoder learning and integrates a gradient weighting mechanism to achieve adapter fusion. Research has found that text semantic information can effectively guide the visual encoder in continuous learning. At the same time, pre-trained text encoders often have difficulty effectively encoding medical terminology when processing natural text data. Therefore, to overcome this problem, this embodiment extracts textual knowledge that is highly aligned with the image content and uses it to guide the learning of the visual encoder.

[0043] In the continuous learning framework TPGCL of this embodiment, the training steps include:

[0044] First, assume that the model will learn C in a new task t t First, the training images x belonging to a specific category in the new task t are i Input image description generation model (such as BLIP) f G , get the corresponding text description f G (x i ). Considering that different diseases may present similar pathological features, we will train the image x i The category name is concatenated with the generated description to enhance the distinction between categories. The final generated format is "This is a {category name} and it is f G (x i )'s disease photo" text description i . Then, using CLIP’s text encoder f T Extract the corresponding text embedding p i =f T (w i ). In order to reduce the individual differences of multiple text embeddings, this embodiment averages all text embeddings of the kth new category to obtain the category text prototype in Represents the set of all image indices belonging to the kth new category.

[0045] These text prototypes guide the visual encoder f through the following CLIP style loss function V Training:

[0046]

[0047] where z n =f V (x n ) represents the training image x n Image encoding, cos(·,·) is the cosine similarity, N t is the total number of training images in task t, yn is x n The category label, is the indicator function.

[0048] In addition to using text prototypes for auxiliary supervision, the image encoding is also fed into the fully connected layer to obtain logits to calculate the conventional cross entropy loss The total loss function is Where λ is a constant coefficient that balances the two losses. It is worth noting that the proposed text prototype can be trained by adding a similar formula The loss term is compatible with baseline methods such as iCaRL, WA, DynamicER, and DarkER++.

[0049] Secondly, this embodiment proposes a gradient weighting mechanism to merge task-specific adapters into a unified task-sharing adapter.

[0050] like Figure 1 As shown in (c), each learnable adapter is designed to include a structure of two fully connected layers and activation functions, and is inserted into each layer of the visual encoder. Let W t，l Represents the loss function by minimizing task t The importance of the task-specific adapter parameters fine-tuned at layer l can be expressed through the loss function For parameter W t，l Gradient To estimate. Since the negative gradient elements with larger magnitude also represent high importance, this embodiment uses the gradient matrix A t，l Square all elements to get A′ t，l , where A′ t，l The value of each element in the matrix represents the adapter parameter in the same position for the newly learned C in the current task t. t The importance of the category, the larger the value, the more important this parameter is. Subsequently, all task-specific adapter parameters {W 1，l ,...,W t，l} are fused by the following formula:

[0051]

[0052] Among them, ⊙ represents the element-by-element product, and both the numerator and denominator are element-by-element operations. s，l represents the gradient matrix of the adapter for the s-th learning task at the l-th layer, A′ j，l The gradient matrix of the adapter for the jth learning task at layer l, W l Will be used as parameters for the shared adapter of layer l tasks.

[0053] By taking this weighted sum over all learned tasks, the parameters that are more important for any particular task will remain relatively stable, thus better preserving the knowledge learned for that task. l Will be used as parameters for the shared adapter of layer l tasks.

[0054] It should be noted that the adapter fusion process may slightly destroy the alignment between the visual encoder output and the classification head. To restore the model performance, this example adopts the same approach as the previous study, sampling a balanced subset of categories from the new and old data, and minimizing the loss Fine-tune the task-sharing adapter. The fine-tuned adapter parameters will also be used to initialize the adapters of subsequent continuous learning tasks.

[0055] During inference, the image description generation module and the text encoder are frozen, and the frozen text encoder and visual encoder are jointly used to recognize test images from any of the learned t tasks.

[0056] Specifically: Figure 1 As shown in (b), the test image is encoded by the adapted visual encoder, the logits output by the fully connected layer 'FC', and the logits obtained by aligning the image encoding and the text prototype are respectively input into the softmax operation, and the two probability vectors are summed and the maximum value corresponding to the category is taken as the prediction result.

[0057] In this example, all experiments were conducted based on CLIP's ViT-B / 16 visual encoder and text encoder architecture. BLIP2-opt-2.7b was used by default as the image description generation model, and other description models (LLaVa-v1.5-7b and Quilt-LLaVa-v1.5-7b) were also tested in the generalization study. During model training, the adapter was updated for 20 training cycles using the Adam optimizer with a cosine annealing strategy, with a batch size of 32 and an initial learning rate of 0.001. The default hidden layer dimension of the adapter was set to 256, and the loss balance coefficient λ was set to 1. The playback sample capacity was configured to 80 on the Skin40 dataset, 32 on Path16, and 16 on Skin8.

[0058] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0059] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0060] The above embodiments are preferred implementations of the present invention, but the implementations of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A medical image continuous learning classification method based on text prototype guidance, characterized by: include: Model training phase: Build a continuous learning framework, including an image description generation module, a text encoder, and a visual encoder; In a continuous learning framework, we generate category text prototypes corresponding to a set of images of a specific category, use the category text prototypes as guidance signals for visual encoder learning, and integrate a gradient weighting mechanism to achieve adapter fusion; Reasoning stage: Freeze the image description generation module and text encoder, input the test image set into the visual encoder that has completed adapter fusion, obtain the image encoding, align the category text prototype with the image encoding, obtain the first value, pass the image encoding through the fully connected layer to output the second value, normalize the first value and the second value respectively, and obtain the prediction result after calculation.

2. The method for continuous learning and classification of medical images based on text prototype guidance according to claim 1, characterized in that: Generating a category text prototype corresponding to a specific category of image sets includes: During the learning task, the image description generation module is used to process the image set of a specific category to obtain the corresponding text description. The category name of the specific category image is spliced ​​with the corresponding text description to obtain the enhanced text description. The text encoder is used to extract the text embedding corresponding to the enhanced text description, and all text embeddings are dedifferentiated to obtain the category text prototype.

3. The method for continuous learning and classification of medical images based on text prototype guidance according to claim 2, characterized in that: All texts are embedded and dedifferentiated as follows: Average all text embeddings of the kth new category to obtain the category text prototype in Represents the set of all image indices belonging to the kth new category.

4. The method for continuous learning and classification of medical images based on text prototype guidance according to claim 1, characterized in that: The method of using the category text prototype as a guiding signal for visual encoder learning includes: Use CLIP style loss function to guide the visual encoder f V The training is as follows Among them, z n =f V (x n ) represents the training image x n Image encoding, cos(·,·) is the cosine similarity, N t is the total number of training images in the task, y n is x n The category label, is the indicator function.

5. The method for continuous learning and classification of medical images based on text prototype guidance according to claim 4 is characterized in that: It also includes inputting the image encoding into the fully connected layer, obtaining logits and calculating the cross entropy loss The total loss of the visual encoder training is Where λ is a constant coefficient that balances the two losses.

6. The method for continuous learning and classification of medical images based on text prototype guidance according to claim 1, characterized in that: The visual encoder learning includes multiple adapters, each adapter includes multiple fully connected layers and activation functions; the learnable adapter is set in each layer of the visual encoder.

7. The method for continuous learning and classification of medical images based on text prototype guidance according to claim 6, characterized in that: The integrated gradient weighting mechanism achieves adapter fusion, including: Importance evaluation: Calculate the gradient of the adapter parameters by setting the loss function to obtain the first gradient matrix, square all elements of the first gradient matrix to obtain the second gradient matrix; Adapter fusion: All adapter parameters {W 1,l ,...,W t,l } to merge and obtain the task sharing adapter as follows: Among them, ⊙ represents the element-by-element product, and both the numerator and denominator are element-by-element operations. s，l represents the gradient matrix of the adapter for the s-th learning task at the l-th layer, A′ j,l The gradient matrix of the adapter for the jth learning task at layer l, W l Will be used as parameters for the shared adapter of layer l tasks.

8. The method for continuous learning and classification of medical images based on text prototype guidance according to claim 7, characterized in that: The gradient of the adapter parameters is calculated as follows: in, represents the loss function, W t，l Represents adapter parameters.

9. The method for continuous learning and classification of medical images based on text prototype guidance according to claim 7, characterized in that: Also includes: Sampling class-balanced subsets from new and old data by minimizing the loss Fine-tune the task sharing adapter.

10. The method for continuous learning and classification of medical images based on text prototype guidance according to claim 1, characterized in that: The first value and the second value are normalized and calculated to obtain the prediction result, specifically: Perform softmax operations on the first and second values ​​respectively, sum the obtained probability vectors, and take the category corresponding to the maximum value as the prediction result.

Images