Robust distillation method and system for medical visual language large model and medium

Abstract

The invention discloses a robust distillation method and system for a medical visual language large model and a medium, and relates to the technical field of computers, and the method comprises the steps: S1, building an image classification training data set, and selecting a teacher model and a random initialization lightweight student model; s2, constructing gradient calibration loss between the teacher model and the student model based on the teacher model gradient; s3, constructing a dynamic gradient alignment loss between the teacher model and the student model based on the sample and category level; s4, constructing hierarchical disturbance feature alignment loss between the teacher model and the student model based on the intermediate disturbance sample; and S5, finely adjusting the image classification model based on double-level weighted gradient alignment loss, level disturbance feature alignment loss, clean sample cross entropy loss, confrontation sample cross entropy loss and standard knowledge distillation loss. And two original conflicting targets of precision keeping and robustness enhancement are decoupled and cooperatively realized, so that the distillation robustness accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a robust distillation method, system and medium for a large medical visual language model. Background Art

[0002] In recent years, large-scale pre-trained visual language models (VLMs) have made significant breakthroughs in cross-modal semantic understanding by aligning visual and textual data into a shared embedding space. Building on these advances, medical visual language foundation models have also achieved significant improvements in fields such as pathology and X-ray imaging. These models utilize large-scale comparative pre-training techniques to capture complex patterns in medical visual and textual data, effectively adapting to downstream medical tasks. Because these models have a large number of parameters and computational overhead, in order to achieve efficient inference of such models in clinical environments and reduce computing resource requirements, knowledge distillation (KD) technology can significantly reduce the computational burden while providing performance close to that of the original large model by transferring the knowledge of large teacher models to lightweight models.

[0003] However, deploying such models in safety-critical applications remains challenging due to their susceptibility to adversarial attacks—malicious perturbations that can disrupt image-text alignment, posing significant security risks. Traditional knowledge distillation methods rely solely on natural examples to maintain the accuracy of the teacher model, neglecting adversarial robustness. This oversight makes the student model vulnerable to adversarial examples, compromising its reliability. This problem is further exacerbated when knowledge is distilled from the base model's complex, high-dimensional feature space. While adversarial distillation (AD) has proven effective in enhancing robustness, the limited capacity of the student model often leads to a trade-off between accuracy and robustness, resulting in suboptimal performance on both fronts. However, these traditional methods rely on a pre-trained robust teacher model, which can compromise the generalization ability of the large base model, and focus solely on output-level alignment, neglecting the teacher model's complex decision signals. This limitation hinders the student model's ability to capture high-level semantic nuances and weakens its overall robustness. Therefore, distilling knowledge from large visual-language models (VLMs) into robust, lightweight models remains an open challenge for safe clinical deployment.

[0004] In view of this, a robust distillation method, system and medium for large medical visual language models were developed to solve the above technical problems. Summary of the Invention

[0005] The purpose of the present invention is to disclose a robust distillation method, system and medium for a large model of medical visual language.

[0006] The first object of the present invention is to provide a robust distillation method for a large medical visual language model.

[0007] The second inventive object of the present invention is to provide a robust distillation system for large medical visual language models.

[0008] A third object of the present invention is to provide a computer-readable storage medium.

[0009] To achieve the above-mentioned first invention objective, the present invention provides a robust distillation method for a large medical visual language model, comprising the following steps: Step S1: establishing an image classification training dataset, selecting a teacher model of a pre-trained large visual language model and a randomly initialized lightweight student model;

[0010] Step S2: Construct the gradient calibration loss between the teacher model and the student model based on the teacher model gradient to obtain the clean sample cross entropy loss function;

[0011] Step S3: Construct a dynamic gradient alignment loss between the teacher model and the student model based on the sample and category levels to obtain the adversarial sample cross entropy loss function and the two-level weighted gradient alignment loss function;

[0012] Step S4: constructing the hierarchical perturbation feature alignment loss between the teacher model and the student model based on the intermediate perturbation samples to obtain the hierarchical perturbation feature alignment loss function;

[0013] Step S5: Based on the two-level weighted gradient alignment loss function, the level perturbation feature alignment loss function, the clean sample cross entropy loss function, the adversarial sample cross entropy loss function and the standard knowledge distillation loss function, the image classification model is comprehensively fine-tuned to obtain the distilled lightweight image classification model.

[0014] Preferably, the training dataset is the pathology dataset CRC100K or the X-ray dataset RSNA;

[0015] The teacher model uses the pre-trained basic model Conch, and the initialized lightweight student model uses the ResNet-18 architecture as the visual encoder.

[0016] Preferably, the training data set is represented as where x i represents a training image, y i Represents its classification label, N is the number of images in the training dataset;

[0017] The teacher model T in step S1 contains a large image encoder E for visual-text alignment img and text encoder E txt ;

[0018] Image encoder of teacher model T The image encoder parameters are denoted as θ T, the image encoder of the student model S The image encoder parameters are denoted as θ S ;

[0019] During the distillation process, keep the text encoder E txt The structure and parameters of are fixed;

[0020] The image encoder parameters are denoted by θ T The teacher model T is used for initialization, and the image encoder parameters are recorded as θ during the distillation process. T Keeping the parameters unchanged, the image encoder parameters θ S Use random initialization.

[0021] Preferably, in step S2, x represents a clean input image, i.e., an image that has not been subjected to adversarial interference, and for the text prompt p of the cth category c , using a fixed text encoder E txt Get text embedding E txt (p c ), input x into the image encoder of the teacher model T and the image encoder of the student model S and obtain visual features;

[0022] By calculating the text embedding E txt (p c ) and the similarity of the visual features of the teacher model T, and through the softmax function, the probability that the teacher model T predicts x as the cth category is obtained

[0023]

[0024] By calculating the text embedding E txt (p c ) and the similarity of the visual features of the student model S, and through the softmax function, the probability that the student model S predicts x as the cth category is obtained

[0025]

[0026] Among them, sim() represents the cosine similarity between two vectors, and P T (x) and P S (x) represents the probability distribution of the teacher model and the student model for all categories predicted on x;

[0027] Use L ce (P T (x), y) represents the cross entropy loss function between the prediction result of the teacher model and the label gold standard y, and is expressed as Indicates the prediction results of the teacher model relative to θ T gradient;

[0028] use Indicates the prediction results of the student model relative to θ S To minimize the gradient difference between the student model and the teacher model, we define the gradient alignment loss L PAG for:

[0029]

[0030] Preferably, in step S3, the alignment loss L PAG Based on the prediction confidence of the teacher model, a dynamic weight α(x) is defined for each input x:

[0031]

[0032] in, is the probability that x belongs to category c predicted by the teacher model, that is, the confidence of the category. At the category level, the category-specific weight τ is dynamically adjusted during the training process. c , to adaptively assign greater weights to more vulnerable categories, first, use Represents the sample after perturbation of x, that is where δ is the image-specific adversarial noise, and represents the cross entropy loss function between the prediction results of the student model and the label gold standard y, and secondly, using R S，c represents the average cross entropy loss of the student model on all adversarial examples in category c:

[0033]

[0034] in, is the sample set of category c in the training data set, in R S，c Based on this, the average value of all categories is calculated to obtain the category average adversarial loss:

[0035]

[0036] use represents the weight of category c at time t, and its update rule is:

[0037]

[0038] Where γ∈(0,1) is the scaling factor. Finally, according to the sample-level weight α(x) and the category-level weight τ c , define the two-level weighted gradient alignment loss as:

[0039]

[0040] Where C is the total number of categories, represents the sample set of category c in the training dataset, Used to balance the influence of category-level weights.

[0041] Preferably, in step S4, represents the adversarial sample after perturbation of x, which is defined as:

[0042]

[0043] Where δ is the perturbation applied to the clean input, and the optimal perturbation value obtained by formula 9 is denoted as δ * , in obtaining Based on δ * In x and Insert L-1 samples between as multi-level adversarial samples, and use represents the adversarial sample of level l, which is defined as:

[0044]

[0045] use represents a set of L multi-level adversarial samples generated by gradient attack, where l represents the perturbation intensity level, is the disturbance sample corresponding to the level condition;

[0046] Secondly, the features of these adversarial samples of varying strengths are aligned with clean inputs to promote robust representation learning. To balance the contributions of adversarial samples of varying strengths during training, a time-varying weighting mechanism is introduced. The Hierarchical Perturbation Feature Alignment (HPFA) loss is defined as:

[0047]

[0048] in represents the image encoder of the student model, w l，t is the adversarial sample of the t-th training cycle The weight follows a gradual transition strategy from weak perturbation to strong perturbation. In the initial stage, a larger weight is given to mild adversarial samples to achieve stable feature learning, and then the weight of strong perturbation samples is gradually increased to achieve precise boundary optimization. The calculation formula is:

[0049]

[0050] Where t max represents the total number of training cycles, and β is a hyperparameter that controls the weight transition rate.

[0051] Preferably, in step S5, the standard knowledge distillation loss L KL , which is defined as:

[0052]

[0053] Where KL represents Kullback-Leibler divergence;

[0054] The total loss function is recorded as:

[0055]

[0056] Where ω1 and ω2 are two weight coefficients to balance the influence of each individual loss function.

[0057] Based on the same inventive principle, to achieve the second invention objective, the present invention provides a robust distillation system for a large medical visual language model, comprising an image classification training dataset, a teacher model for pre-training the large visual language model, and a randomly initialized lightweight student model.

[0058] By inputting a clean image x into the teacher model, we get the clean sample cross entropy loss function between the teacher model's prediction result and the label gold standard y;

[0059] By inputting an adversarial image into the student model Solve the adversarial cross entropy loss function between the student model’s predictions and the label ground truth y;

[0060] Define a two-level weighted gradient alignment loss function, a level-perturbation feature alignment loss function, and a standard knowledge distillation loss function to comprehensively fine-tune the image classification model;

[0061] The image classification model is fine-tuned based on the two-level weighted gradient alignment loss function, the hierarchical perturbation feature alignment loss function, the clean sample cross entropy loss function, the adversarial sample cross entropy loss function and the standard knowledge distillation loss function to obtain a lightweight image classification model after distillation.

[0062] Preferably, the training dataset is the pathology dataset CRC100K or the X-ray dataset RSNA;

[0063] The teacher model uses the pre-trained basic model Conch, and the initialized lightweight student model uses the ResNet-18 architecture as the visual encoder.

[0064] Based on the same inventive principle, in order to achieve the above-mentioned third invention purpose, the present invention provides a computer-readable storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, it implements the robust distillation method of the medical visual language large model as described in the first invention.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] (1) The gradient of the teacher model is used as a "perceptual guide" to directly guide the student model to learn the sensitivity of the decision boundary; the dynamic gradient calibration mechanism utilizes the perceptually guided gradient to eliminate the need for a pre-trained robust teacher model while maintaining generalization ability; by designing a two-level weighted gradient alignment loss function to dynamically adjust the priority of gradient alignment, the student model is prevented from being forced to imitate the potentially redundant robust features in the teacher model due to capacity limitations. The dual weighting strategy combines sample adaptive weighting with class-aware gradient coordination to ensure balanced and consistent knowledge transfer between heterogeneous architectures.

[0067] (2) Generate a set of intermediate adversarial samples through multi-level perturbations (rather than just the strongest adversarial samples), covering a continuous range of perturbation intensities from weak to strong; align the features of adversarial samples under different perturbation intensities with the features of clean samples to smooth the local distortion of the feature space and reduce mutations near the decision boundary; prioritize alignment of weakly perturbed samples (stable learning) at the beginning of training, and gradually introduce strong perturbation samples (finely optimize the boundary) in the later stage to avoid overfitting the model to a certain perturbation intensity, thereby improving the overall robustness; the hierarchical adversarial feature alignment strategy uses intermediate adversarial samples to optimize the feature distribution between clean input and perturbation input, thereby further enhancing the robustness of the decision boundary.

[0068] (3) Compared with the traditional distillation method, the present invention comprehensively fine-tunes the image classification model based on the two-level weighted gradient alignment loss function, the hierarchical perturbation feature alignment loss function, the clean sample cross entropy loss function, the adversarial sample cross entropy loss function and the standard knowledge distillation loss function. Through the multi-objective joint optimization mechanism, the two originally conflicting goals of accuracy preservation and robustness enhancement are decoupled and achieved synergistically. It not only maintains competitive performance on clean samples, but also guides the visual encoder to learn features related to robustness, significantly improving the robust accuracy of adversarial distillation and defense methods, thereby taking into account both the prediction accuracy and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 Flowchart of the robust distillation method for the medical visual language large model of the present invention.

[0070] Figure 2 This is a schematic diagram of the robust distillation method for the medical visual language large model of the present invention.

[0071] Figure 3 This is a topological diagram of the robust distillation method for the medical visual language large model of the present invention.

[0072] Figure 4Comparison of the robust accuracy of the present invention with other fine-tuning methods on the CRC100K training dataset.

[0073] Figure 5 The robust accuracy of the present invention on the RSNA training dataset is compared with other fine-tuning methods.

[0074] Figure 6 It is a schematic diagram of the computer medium module of the present invention. DETAILED DESCRIPTION

[0075] The present invention is described in detail below with reference to the various embodiments shown in the accompanying drawings, but it should be noted that these embodiments are not limitations of the present invention, and any equivalent transformations or substitutions in functions, methods, or structures made by ordinary technicians in this field based on these embodiments are all within the scope of protection of the present invention.

[0076] The specific implementation process of the present invention is described below through multiple embodiments.

[0077] Example 1:

[0078] Ginseng Figures 1 to 5 As shown, this embodiment discloses a specific implementation method of a robust distillation method for a large medical visual language model (hereinafter referred to as the "method").

[0079] In this embodiment, this embodiment is implemented in a computer with an Intel(R) Core(TM) i7-6850K CPU @ 3.60GHZ, a GeForce GTX3090 GPU, and 24.0GB of memory. The programming language is Python, and the fine-tuning of the deep neural network is implemented through Pytorch.

[0080] Ginseng Figures 1 to 5 As shown in FIG, the robust distillation method for a large medical visual language model includes the following steps:

[0081] Step S1: Establish an image classification training dataset, select a teacher model of a pre-trained visual language large model and a randomly initialized lightweight student model; specifically, the training dataset is the pathology dataset CRC100K (whose images have 9 categories, 100,000 training images and 7,000 verification images) or the X-ray dataset RSNA (whose images have 2 categories, 26,684 training images and 3,000 verification images); the teacher model uses the pre-trained basic model Conch, and the initialized lightweight student model uses the ResNet-18 architecture as the visual encoder; the ratio of the number of images in the training dataset to the number of images in the test image set is 85:15; the training dataset is represented as where x irepresents a training image, y i represents its classification label, N is the number of images in the training dataset; the teacher model T in step S1 contains a large image encoder E for visual-text alignment img and text encoder E txt ; Image encoder of teacher model T The image encoder parameters are denoted as θ T , the image encoder of the student model S The image encoder parameters are denoted as θ S ; During the distillation process, keep the text encoder E txt The structure and parameters of the image encoder are fixed; the image encoder parameters are denoted by θ T The teacher model T is used for initialization, and the image encoder parameters are recorded as θ during the distillation process. T Keeping the parameters unchanged, the image encoder parameters θ S Use random initialization. Through step S1, freeze the text encoder E txt The text-visual alignment capability is obtained by pre-training large-scale image-text comparison in the teacher model, which is retained intact, saving computing power and avoiding destroying the learned cross-modal semantic space; the image encoder parameter memory θ is frozen. T During the entire distillation stage, it is only used as a "reference answer" and "robust signal source" and is no longer updated. This not only ensures the stability of the teacher model output, but also provides a constant and reliable comparison benchmark for all subsequent losses; the student image encoder is randomly initialized, and the visual representation task that really needs to be "learned" is completely handed over to the lightweight student network, so that in subsequent steps through dynamic gradient calibration, layered adversarial alignment and other mechanisms, the visual knowledge and adversarial robustness of the teacher model are "learned", and finally a lightweight and robust medical image encoder is obtained.

[0082] Step S2: Based on the teacher model gradient, the gradient calibration loss between the teacher model and the student model is constructed to obtain the clean sample cross entropy loss function; Specifically, in step S2, x represents a clean input image, that is, an image that has not been subjected to adversarial interference, and for the text prompt p of the cth category c , using a fixed text encoder E txt Get text embedding E txt (p c ), input x into the image encoder of the teacher model T and the image encoder of the student model S and obtain visual features;

[0083] By calculating the text embedding E txt (p c) and the similarity of the visual features of the teacher model T, and through the softmax function, the probability that the teacher model T predicts x as the cth category is obtained

[0084]

[0085] By calculating the text embedding E txt (p c ) and the similarity of the visual features of the student model S, and through the softmax function, the probability that the student model S predicts x as the cth category is obtained

[0086]

[0087] Among them, sim() represents the cosine similarity between two vectors, and P T (x) and P S (x) represents the probability distribution of the teacher model and the student model for all categories predicted on x;

[0088] Use L ce (P T (x), y) represents the cross entropy loss function between the prediction result of the teacher model and the label gold standard y, and is expressed as Indicates the prediction results of the teacher model relative to θ T gradient;

[0089] use Indicates the prediction results of the student model relative to θ S To minimize the gradient difference between the student model and the teacher model, we define the gradient alignment loss L PAG for:

[0090]

[0091] Through step S2, the "gradient signal" of the teacher model on the clean sample is extracted and used as a "perception guide" to guide the student model to learn decision sensitivity similar to that of the teacher model, so as to obtain robustness without losing accuracy in subsequent adversarial training.

[0092] Step S3: Construct a dynamic gradient alignment loss between the teacher model and the student model based on the sample and category levels to obtain the adversarial sample cross entropy loss function and the two-level weighted gradient alignment loss function; specifically, in step S3, the alignment loss L PAG Based on the prediction confidence of the teacher model, a dynamic weight α(x) is defined for each input x:

[0093]

[0094] in, is the probability that x belongs to category c predicted by the teacher model, that is, the confidence of the category. At the category level, the category-specific weight τ is dynamically adjusted during the training process. c , to adaptively assign greater weights to more vulnerable categories, first, use Represents the sample after perturbation of x, that is where δ is the image-specific adversarial noise, and represents the cross entropy loss function between the prediction results of the student model and the label gold standard y, and secondly, using R S，c represents the average cross entropy loss of the student model on all adversarial examples in category c:

[0095]

[0096] in, is the sample set of category c in the training data set, in R S，c Based on this, the average value of all categories is calculated to obtain the category average adversarial loss:

[0097]

[0098] use represents the weight of category c at time t, and its update rule is:

[0099]

[0100] Where γ∈(0,1) is the scaling factor, which is set to 0.1 in this embodiment. Finally, according to the sample-level weight α(x) and the category-level weight τ c , define the two-level weighted gradient alignment loss as:

[0101]

[0102] Where C is the total number of categories. In this embodiment, C=9 (pathological images) or C=2 (X-ray images). represents the sample set of category c in the training dataset, Used to balance the impact of category-level weights, is the category-level weight coefficient; through step S3, a two-level weighted gradient alignment loss function is designed to dynamically adjust the priority of gradient alignment to prevent the student model from being forced to imitate the potentially redundant robust features in the teacher model due to capacity limitations. The dual-weighted strategy combines sample adaptive weighting with class-aware gradient coordination to ensure balanced and consistent knowledge transfer between heterogeneous architectures; the "sample-category dual-dimensional adaptive weight" is used to dynamically amplify the influence of the "most difficult to learn and most fragile" samples / categories in gradient alignment, thereby solving the problem of individual difficult examples lagging behind and avoiding overfitting or lack of robustness of the model on a few categories, ultimately achieving a subtle improvement in overall robustness.

[0103] Step S4: construct the hierarchical perturbation feature alignment loss between the teacher model and the student model based on the intermediate perturbation samples, and obtain the hierarchical perturbation feature alignment loss function; specifically, in step S4, use represents the adversarial sample after perturbation of x, which is defined as:

[0104]

[0105] Where δ is the perturbation applied to the clean input, and the optimal perturbation value obtained by formula 9 is denoted as δ * , in obtaining Based on δ * In x and Insert L-1 samples between them as multi-level adversarial samples. In this embodiment, L=5. represents the adversarial sample of level l, which is defined as:

[0106]

[0107] use represents a set of L multi-level adversarial samples generated by gradient attack, where l represents the perturbation intensity level, is the disturbance sample corresponding to the level condition;

[0108] Secondly, the features of these adversarial samples of varying strengths are aligned with clean inputs to promote robust representation learning. To balance the contributions of adversarial samples of varying strengths during training, a time-varying weighting mechanism is introduced. The Hierarchical Perturbation Feature Alignment (HPFA) loss is defined as:

[0109]

[0110] in represents the image encoder of the student model, w l，t is the adversarial sample of the t-th training cycle The weight follows a gradual transition strategy from weak perturbation to strong perturbation. In the initial stage, a larger weight is given to mild adversarial samples to achieve stable feature learning, and then the weight of strong perturbation samples is gradually increased to achieve precise boundary optimization. The calculation formula is:

[0111]

[0112] Where, t max =50 represents the total number of training cycles, and β = 2.0 is a hyperparameter that controls the weight transition rate. Through step S4, using "multi-layer perturbation samples from weak to strong" as "course materials", the student model first learns to be insensitive to slight perturbations during training, and then gradually hardens to the boundary of strong perturbations, thereby smoothly pulling the clean feature space and the adversarial space together, and finally obtaining a decision boundary that is both compact and robust.

[0113] Step S5: Based on the two-level weighted gradient alignment loss function, the level perturbation feature alignment loss function, the clean sample cross entropy loss function, the adversarial sample cross entropy loss function and the standard knowledge distillation loss function, the image classification model is fine-tuned to obtain the lightweight image classification model after distillation; specifically, see Figure 3 , in step S5, the standard knowledge distillation loss L KL , which is defined as:

[0114]

[0115] Where KL represents Kullback-Leibler divergence;

[0116] The total loss function is recorded as:

[0117]

[0118] Among them, ω1 and ω2 are two weight coefficients, which are both set to 1.0 in this embodiment to balance the influence of each individual loss function; compared with the traditional distillation method, this embodiment is based on a two-level weighted gradient alignment loss function, a hierarchical perturbation feature alignment loss function, a clean sample cross entropy loss function, an adversarial sample cross entropy loss function and a standard knowledge distillation loss function to comprehensively fine-tune the image classification model. Through a multi-objective joint optimization mechanism, the two originally conflicting goals of accuracy preservation and robustness enhancement are decoupled and achieved in a coordinated manner. It not only maintains competitive performance on clean samples, but also guides the visual encoder to learn features related to robustness, significantly improving the robust accuracy of adversarial distillation and defense methods, thereby taking into account both the prediction accuracy and robustness of the model. During the distillation training stage, this embodiment only trains the visual encoder of the student model for 50 rounds, using an initial learning rate of 0.1, a momentum of 0.9, and a weight decay of 5×10 -4The SGD optimizer is used; the batch size is set to 128, and the internal maximization process adopts a 10-step PGD attack strategy; the initial perturbation amplitude is set to 0.001, the step size is 2 / 255, and the L∞ perturbation boundary ∈=8 / 255.

[0119] See also Figure 4 and Figure 5 We evaluate the performance of a lightweight image classification model (a lightweight student model) by measuring its resistance to adversarial attacks (called robust accuracy). Robust accuracy is measured using two widely used metrics: FGSM and PGD. When calculating these metrics, we set the maximum perturbation amplitude to 8 / 255 and use 20 iterations of PGD, each with a step size of 2 / 255. Both metrics range from 0 to 100%, with higher values ​​indicating better performance.

[0120] Figure 4 This figure compares the FGSM performance of our method with existing methods when distilling the CLIP ViT / B-16 model on the CRC100K and RSNA datasets. RobustWRN and CTRW are traditional regularization-based robust attack defense methods, while TRADES and RSLAD are two existing adversarial distillation methods. As can be seen, after fine-tuning, our method achieves significantly higher FGSM performance than the other methods.

[0121] Figure 5 This is a comparison of the PGD index between the method of the present invention and the existing method when distilling the CLIP ViT / B-16 model on the CRC100K and RSNA datasets. It can also be seen that the method of the present invention has achieved significant robustness improvement compared with the existing robustness training method and distillation method.

[0122] Example 2:

[0123] This embodiment discloses a specific implementation of a robust distillation system for a large medical visual language model (hereinafter referred to as the "system").

[0124] Robust distillation system for large medical visual language models, see Figure 2 and Figure 3 , executing the robust distillation method of the medical visual language large model described in Example 1, including an image classification training dataset, a teacher model of a pre-trained visual language large model, and a randomly initialized lightweight student model; by inputting a clean image x into the teacher model, obtaining a clean sample cross entropy loss function between the teacher model's prediction result and the label gold standard y; by inputting an adversarial image into the student model Solve the adversarial sample cross entropy loss function between the prediction results of the student model and the label gold standard y; define the two-level weighted gradient alignment loss function, the level perturbation feature alignment loss function and the standard knowledge distillation loss function to comprehensively fine-tune the image classification model; based on the two-level weighted gradient alignment loss function, the level perturbation feature alignment loss function, the clean sample cross entropy loss function, the adversarial sample cross entropy loss function and the standard knowledge distillation loss function, the image classification model is comprehensively fine-tuned to obtain the lightweight image classification model after distillation.

[0125] Specifically, the training dataset is the pathology dataset CRC100K (whose images have 9 categories, 100,000 training images and 7,000 verification images) or the X-ray dataset RSNA (whose images have 2 categories, 26,684 training images and 3,000 verification images); the teacher model uses the pre-trained basic model Conch, and the initialized lightweight student model uses the ResNet-18 architecture as the visual encoder; the ratio of the number of images in the training dataset to the number of images in the test image set is 85:15; this embodiment is based on a comprehensive fine-tuning image classification model based on a two-level weighted gradient alignment loss function, a hierarchical perturbation feature alignment loss function, a clean sample cross entropy loss function, an adversarial sample cross entropy loss function and a standard knowledge distillation loss function. Through a multi-objective joint optimization mechanism, the two originally conflicting goals of accuracy maintenance and robustness enhancement are decoupled and synergistically achieved, not only maintaining competitive performance on clean samples, but also guiding the visual encoder to learn features related to robustness, significantly improving the robust accuracy of adversarial distillation and defense methods, thereby taking into account both the prediction accuracy and robustness of the model.

[0126] The robust distillation system of the medical visual language large model disclosed in this embodiment has the same technical solutions as those in Example 1. Please refer to Example 1 and will not repeat them here.

[0127] Example 3:

[0128] Computer readable storage medium, see Figure 6 , which stores a computer program, which, when executed by a processor, implements the robust distillation method of the medical visual language large model as described in Example 1.

[0129] The computer-readable storage medium disclosed in this embodiment has the same technical solutions as those in the first embodiment. Please refer to the first embodiment and will not be repeated here.

[0130] The various illustrative logic blocks or units described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0131] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

[0132] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

[0133] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A robust distillation method for large medical visual language models, characterized by: The following steps are involved: Step S1: Create an image classification training dataset, select a teacher model of a pre-trained visual language model and a randomly initialized lightweight student model; Step S2: Construct the gradient calibration loss between the teacher model and the student model based on the teacher model gradient to obtain the clean sample cross entropy loss function; Step S3: Construct a dynamic gradient alignment loss between the teacher model and the student model based on the sample and category levels to obtain the adversarial sample cross entropy loss function and the two-level weighted gradient alignment loss function; Step S4: constructing the hierarchical perturbation feature alignment loss between the teacher model and the student model based on the intermediate perturbation samples to obtain the hierarchical perturbation feature alignment loss function; Step S5: Based on the two-level weighted gradient alignment loss function, the level perturbation feature alignment loss function, the clean sample cross entropy loss function, the adversarial sample cross entropy loss function and the standard knowledge distillation loss function, the image classification model is comprehensively fine-tuned to obtain the distilled lightweight image classification model.

2. The robust distillation method for a large medical visual language model according to claim 1, characterized in that: The training dataset is the pathology dataset CRC100K or the X-ray dataset RSNA; The teacher model uses the pre-trained basic model Conch, and the initialized lightweight student model uses the ResNet-18 architecture as the visual encoder.

3. The robust distillation method for a large medical visual language model according to claim 1 or 2, characterized in that: The training dataset is represented as where x i represents a training image, y i Represents its classification label, N is the number of images in the training dataset; The teacher model T in step S1 contains a large image encoder E for visual-text alignment img and text encoder E txt ; Image encoder of teacher model T The image encoder parameters are denoted as θ T , the image encoder of the student model S The image encoder parameters are denoted as θ S ; During the distillation process, keep the text encoder E txt The structure and parameters of are fixed; The image encoder parameters are denoted by θ T The teacher model T is used for initialization, and the image encoder parameters are recorded as θ during the distillation process. T Keeping the parameters unchanged, the image encoder parameters θ S Use random initialization.

4. The robust distillation method for a large medical visual language model according to claim 3, wherein: In step S2, let x denote a clean input image, i.e., an image that has not been subjected to adversarial interference, and for the text prompt p of the cth category c , using a fixed text encoder E txt Get text embedding E txt (p c ), input x into the image encoder of the teacher model T and the image encoder of the student model S and obtain visual features; By calculating the text embedding E txt (p c ) and the similarity of the visual features of the teacher model T, and through the softmax function, the probability that the teacher model T predicts x as the cth category is obtained By calculating the text embedding E txt (p c ) and the similarity of the visual features of the student model S, and through the softmax function, the probability that the student model S predicts x as the cth category is obtained Among them, sim() represents the cosine similarity between two vectors, and P T (x) and P S (x) represents the probability distribution of the teacher model and the student model for all categories predicted on x; Use L ce (P T (x), y) represents the cross entropy loss function between the prediction result of the teacher model and the label gold standard y, and is expressed as Indicates the prediction results of the teacher model relative to θ T gradient; use Indicates the prediction results of the student model relative to θ S To minimize the gradient difference between the student model and the teacher model, we define the gradient alignment loss L PAG for:

5. The robust distillation method for a large medical visual language model according to claim 4, characterized in that: In step S3, the alignment loss L PAG Based on the prediction confidence of the teacher model, a dynamic weight α(x) is defined for each input x: in, is the probability that x belongs to category c predicted by the teacher model, that is, the confidence of the category. At the category level, the category-specific weight τ is dynamically adjusted during the training process. c , to adaptively assign greater weights to more vulnerable categories, first, use Represents the sample after perturbation of x, that is where δ is the image-specific adversarial noise, and represents the cross entropy loss function between the prediction results of the student model and the label gold standard y, and secondly, using R S,c represents the average cross entropy loss of the student model on all adversarial examples in category c: in, is the sample set of category c in the training data set, in R S，c Based on this, the average value of all categories is calculated to obtain the category average adversarial loss: use represents the weight of category c at time t, and its update rule is: Where γ∈(0,1) is the scaling factor. Finally, according to the sample-level weight α(x) and the category-level weight τ c , define the two-level weighted gradient alignment loss as: Where C is the total number of categories, represents the sample set of category c in the training dataset, Used to balance the influence of category-level weights.

6. The robust distillation method for a large medical visual language model according to claim 5, characterized in that: In step S4, represents the adversarial sample after perturbation of x, which is defined as: Where δ is the perturbation applied to the clean input, and the optimal perturbation value obtained by formula 9 is denoted as δ * , in obtaining Based on δ * In x and Insert L-1 samples between as multi-level adversarial samples, and use represents the adversarial sample of level l, which is defined as: use represents a set of L multi-level adversarial samples generated by gradient attack, where l represents the perturbation intensity level, is the disturbance sample corresponding to the level condition; Secondly, the features of these adversarial samples of varying strengths are aligned with clean inputs to promote robust representation learning. To balance the contributions of adversarial samples of varying strengths during training, a time-varying weighting mechanism is introduced. The Hierarchical Perturbation Feature Alignment (HPFA) loss is defined as: in represents the image encoder of the student model, w l，t is the adversarial sample of the t-th training cycle The weight follows a gradual transition strategy from weak perturbation to strong perturbation. In the initial stage, a larger weight is given to mild adversarial samples to achieve stable feature learning, and then the weight of strong perturbation samples is gradually increased to achieve precise boundary optimization. The calculation formula is: Where t max represents the total number of training cycles, and β is a hyperparameter that controls the weight transition rate.

7. The robust distillation method for a large medical visual language model according to claim 6, wherein: In step S5, the standard knowledge distillation loss L KL , which is defined as: Where KL represents Kullback-Leibler divergence; The total loss function is recorded as: Where ω1 and ω2 are two weight coefficients to balance the influence of each individual loss function.

8. Robust distillation system for large medical visual language models, characterized by: Includes image classification training dataset, pre-trained visual language model teacher model and randomly initialized lightweight student model; By inputting a clean image x into the teacher model, we get the clean sample cross entropy loss function between the teacher model's prediction result and the label gold standard y; By inputting an adversarial image into the student model Solve the adversarial cross entropy loss function between the student model’s predictions and the label ground truth y; Define a two-level weighted gradient alignment loss function, a level-by-level perturbation feature alignment loss function, and a standard knowledge distillation loss function to comprehensively fine-tune the image classification model; The image classification model is fine-tuned based on the two-level weighted gradient alignment loss function, the hierarchical perturbation feature alignment loss function, the clean sample cross entropy loss function, the adversarial sample cross entropy loss function and the standard knowledge distillation loss function to obtain a lightweight image classification model after distillation.

9. The robust distillation system for a large medical visual language model according to claim 8, characterized in that: The training dataset is the pathology dataset CRC100K or the X-ray dataset RSNA; The teacher model uses the pre-trained basic model Conch, and the initialized lightweight student model uses the ResNet-18 architecture as the visual encoder.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the robust distillation method of the medical visual language large model as described in any one of claims 1 to 7 is implemented.

Images