Unified anomaly detection method and system based on conditional adapter

Abstract

The invention provides a unified anomaly detection method and system based on a conditional adapter, and the method comprises the steps: generating a field-related text prompt set through a large-scale language model, implanting LoRA adapters in a visual encoder and a text encoder, freezing original parameters, freezing the visual encoder in the first stage, and freezing the text encoder in the second stage; in the first stage, text features and visual representation of a medical image are extracted, a weighted contrast loss training text encoder LoRA is constructed, text anchor point features are generated, in the second stage, the text encoder is frozen, weighted contrast loss is multiplexed to drive the visual encoder LoRA to train, and conditional alignment of the visual features and text anchor points is achieved. And finally performing classification and segmentation tasks on the test image at the same time. According to the method, the two processes of'text anchor point learning 'and'visual feature alignment' are completely decoupled, so that the optimization target of each stage becomes single and clear, the stability and the high efficiency of the training process are ensured, and a global optimal solution is more easily converged.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and in particular to a unified anomaly detection method and system based on a conditional adapter. Background Art

[0002] The present invention relates to the field of computer vision and artificial intelligence technologies, and more specifically, to a two-dimensional (2D) image anomaly detection and segmentation technology for use in industrial manufacturing, medical diagnosis, and other scenarios.

[0003] Anomaly detection is a crucial task in modern production and quality control processes. It aims to identify abnormal or defective samples that differ significantly from normal samples. Traditional supervised learning methods typically require a large number of accurately labeled normal and abnormal samples for model training. However, in practical applications, the low probability of abnormal samples makes data collection difficult and labeling expensive, severely restricting the application of supervised learning methods.

[0004] In recent years, large-scale vision-language pre-training models (VLMs), exemplified by CLIP (Contrastive Language-Image Pre-training), have demonstrated remarkable zero-shot learning capabilities across a wide range of downstream tasks. These models employ contrastive learning on massive amounts of image-text pairs, enabling their encoders to generate a semantically aligned, highly generalized multimodal feature space. This provides a novel and highly promising technical approach to solving the zero-shot anomaly detection problem. Therefore, effectively utilizing and adapting foundational models like CLIP to not only perform global image-level anomaly detection but also achieve refined pixel-level anomaly localization and segmentation remains a critical and pressing challenge.

[0005] This paper aims to address two core pain points faced by current vision-language pre-training models in the field of fine-grained anomaly detection: First, existing methods often fine-tune independently for single attribute scenarios (e.g., “normal / abnormal”), resulting in numerous models, redundant parameters, high deployment costs, and inflexible scalability. Secondly, it relies heavily on manual prompt engineering and fixed adapters, which not only lacks the ability to universally perceive arbitrary attribute dimensions, but also makes it difficult to dynamically inject new attribute knowledge according to user needs during runtime, thus limiting the real-time and maintainability of the model in applications such as industrial inspection, medical imaging, and multimodal search. Summary of the Invention

[0006] In view of the above situation, the main purpose of the present invention is to propose a unified anomaly detection method and system based on conditional adapters to solve the above technical problems.

[0007] The present invention proposes a unified anomaly detection method based on a conditional adapter, the method comprising the following steps: Step 1: Obtain domain-related seed questions and input them into a large language model for processing to obtain a text prompt set containing normal text prompts and abnormal text prompts; Step 2: Load the pre-trained vision-language model including the visual encoder and the text encoder, implant the LoRA adapter into the visual encoder and the text encoder respectively, and freeze the original weight parameters of the visual encoder and the text encoder to obtain the fine-tuned vision-language model; Step 3. In the first stage, the LoRA adapter embedded in the text encoder is set to a trainable state, and the fine-tuned vision-language model is used to extract features from the medical image and the text prompt set to obtain visual representations and text features, respectively. A weighted contrastive learning loss function for the first stage is constructed based on the visual representations and text features, and the text encoder is trained using the weighted contrastive learning loss function for the first stage to obtain the text encoder after the first stage training. Step 4. In the second stage, the LoRA adapter implanted in the visual encoder is set to a trainable state, and the text encoder trained in the first stage is used to extract features of the text prompt set to obtain text anchor features. The weighted contrast learning loss function of the second stage is constructed based on the visual representation and the text anchor features. The visual encoder is trained using the weighted contrast learning loss function of the second stage to obtain the visual encoder trained in the second stage. The text encoder trained in the first stage and the visual encoder trained in the second stage are used to form a trained vision-language model. Step 5: Input the image to be tested, use the trained vision-language model to extract features from the image to be tested and the text prompt set, and construct classification and segmentation tasks respectively to obtain image-level anomaly scores and anomaly segmentation maps.

[0008] The present invention further provides a unified anomaly detection system based on a conditional adapter, wherein the system applies the unified anomaly detection method based on a conditional adapter as described above, and the system includes: Prompt library building blocks for: Obtain domain-related seed questions and input them into a large language model for processing to obtain a text prompt set containing normal text prompts and abnormal text prompts; Adapter implant module for: Load the pre-trained vision-language model containing the visual encoder and text encoder, implant the LoRA adapter into the visual encoder and text encoder respectively, and freeze the original weight parameters of the visual encoder and text encoder to obtain the fine-tuned vision-language model; Two-stage training module for: In the first stage, the LoRA adapter embedded in the text encoder is set to a trainable state, and a fine-tuned vision-language model is used to extract features from medical images and text prompt sets to obtain visual representations and text features, respectively. A weighted contrastive learning loss function is constructed based on the visual representations and text features, and the text encoder is trained using the weighted contrastive learning loss function to obtain the text encoder after the first stage training. In the second stage, the LoRA adapter embedded in the visual encoder is set to a trainable state. The text encoder trained in the first stage is used to extract features from the text prompt set to obtain text anchor features. The weighted contrastive learning loss function of the second stage is constructed based on the visual representation and the text anchor features. The visual encoder is trained using the weighted contrastive learning loss function of the second stage to obtain the visual encoder trained in the second stage. The text encoder trained in the first stage and the visual encoder trained in the second stage are used to form a trained vision-language model. Anomaly detection module, used to: Input the image to be tested, use the trained vision-language model to extract features from the image to be tested and the text prompt set, and construct classification and segmentation tasks respectively to obtain image-level anomaly scores and anomaly segmentation maps respectively.

[0009] Compared with the prior art, the present invention has the following beneficial effects: 1. Higher detection accuracy and stronger generalization capability. This technical effect is mainly due to the deep innovations of the present invention in two aspects: semantic understanding and feature space construction. Unlike the existing technology that only uses general learnable text of "normal / abnormal", the present invention uses a large language model to construct a semantic world with rich details and layers. This enables the model to learn and understand diverse and specific abnormal forms, rather than just the general concept of "abnormal". When faced with new and unseen anomalies, this refined semantic guidance capability enables the present invention to have zero-shot generalization performance far exceeding that of the existing technology. In addition, the weighted contrastive learning loss function adopted by the present invention constructs a feature space with clearer boundaries and higher discrimination by dynamically weighting the gradients of difficult negative samples. 2. Stronger adaptability of visual features and more refined segmentation capabilities: This invention addresses the fundamental flaw of the existing technology, where the visual encoder is completely frozen and cannot be optimized for downstream tasks. By implanting and specifically training the LoRA adapter in the visual encoder during the second stage of training, the model's "visual perception" ability can be adaptively fine-tuned according to specific anomaly detection tasks. The model can therefore learn to actively focus on key visual cues such as micro-texture and morphology related to specific anomalies. 3. The lightweight multi-scale visual feature fusion module designed in this invention can adaptively fuse shallow, medium, and deep features from the visual encoder, simultaneously capturing local details and global context. Compared with the existing method of extracting local features independently from each layer and generating segmentation maps through simple merging and post-processing, this module can generate visual representations with more complete information and more accurate boundary perception. As a result, this invention demonstrates significantly better precision and accuracy in the tasks of locating and pixel-level segmentation of abnormal areas, especially when dealing with small, low-contrast, or irregularly shaped anomalies. 4. A more stable and efficient training process and better convergence results; the original two-stage conditional training strategy of the present invention fundamentally solves the optimization goal conflict problem existing in the existing technology in single-stage training. The existing technology attempts to use the same set of learnable text vectors to align global image features and multi-level local image features at the same time, which can easily lead to unstable training process and convergence to suboptimal solutions; the present invention completely decouples the two processes of "text anchor learning" and "visual feature alignment", making the optimization goals of each stage single and clear, thereby ensuring the stability and efficiency of the training process and making it easier to converge to a global optimal solution. At the same time, the LoRA adapter used in the present invention is a specific implementation method for efficient parameter fine-tuning. Compared with the method of only updating the text prompt vector in the existing technology, it is a fine-tuning paradigm with a wider application range and generally stronger performance. Therefore, while achieving higher performance, the present invention also ensures high efficiency and low resource consumption of training.

[0010] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a flowchart of the steps of a unified anomaly detection method based on conditional adapter proposed by the present invention; Figure 2 This is the overall framework diagram of a unified anomaly detection method based on conditional adapter proposed in the present invention; Figure 3 This is a system structure diagram of a unified anomaly detection system based on conditional adapters proposed by the present invention. DETAILED DESCRIPTION

[0012] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0013] These and other aspects of the embodiments of the present invention will become clear with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0014] See also Figure 1 and Figure 2 This embodiment provides a unified anomaly detection method based on a conditional adapter, the method comprising the following steps: Step 1: Obtain domain-related seed questions and input them into a large language model for processing to obtain a text prompt set containing normal text prompts and abnormal text prompts.

[0015] It should be noted that in Figure 2 middle, represents a large language model, All represent Transformer modules. Both represent adapter modules. represents element-wise multiplication, Represents the projection module, Represents the fused overall text features output by the text encoder, represents the text features for the anomaly classification task, represents the text features for the anomaly segmentation task, represents the visual features for the anomaly classification task, represents the visual features for the anomaly segmentation task, represents feature addition, represents the abnormal segmentation task, represents anomaly classification task; The "+" symbol represents the positive class. In the field of anomaly detection technology involved in this invention, the positive class specifically refers to the "abnormal" state. In the label {-, +} of the anomaly classification task (AC), the "+" symbol indicates that the ground truth value of the image is an anomaly sample; in the label of the anomaly segmentation task (AS), the "+" symbol indicates that a pixel in the pixel-level ground truth mask belongs to the anomaly region.

[0016] The symbol "-": This symbol represents the negative class. In the technical context of this invention, the negative class specifically refers to the "normal" state. In the label {-, +} of the anomaly classification (AC) task, the symbol "-" indicates that the ground truth value of the image is a normal sample; in the label of the anomaly segmentation (AS) task, the symbol "-" indicates that a pixel point in the pixel-level truth mask belongs to the normal area; Figure 2 The two emoticons used in the text are mnemonics that serve as an auxiliary tool to enhance intuitive understanding. Their purpose is to further clarify the semantic connotations of positive and negative categories. Their specific meanings are as follows: The emoticon icon "Happy Face": This icon points to the "+" symbol. In the anomaly detection task performed by the present invention, successfully finding the target to be detected, that is, the "anomaly", is considered a successful task achievement. Therefore, the "Happy Face" is used to refer to the positive class (i.e., the "normal" state); Emoticon icon "Sad expression": This icon points to the symbol "-". In the task context of the present invention, no abnormal target was found, corresponding to a result that the detection purpose was not achieved. Therefore, the "sad expression" is used to refer to the negative class (Negative Class), that is, the "abnormal" state.

[0017] Step 2: Load the pre-trained vision-language model containing the visual encoder and text encoder, implant the LoRA adapter into the visual encoder and text encoder respectively, and freeze the original weight parameters of the visual encoder and text encoder to obtain the fine-tuned vision-language model.

[0018] In step 2, a pre-trained vision-language model including a visual encoder and a text encoder is loaded, the LoRA adapter is implanted into the visual encoder and the text encoder respectively, and the original weight parameters of the visual encoder and the text encoder are frozen to obtain a fine-tuned vision-language model. The specific steps include the following: Load the pre-trained vision-language model containing the visual encoder and text encoder, implant the LoRA adapter into the visual encoder, and freeze the original weight parameters of the visual encoder to obtain the fine-tuned visual encoder; The LoRA adapter is implanted into the text encoder, and the original weight parameters of the text encoder are frozen to obtain a fine-tuned text encoder; A fine-tuned vision-language model is constructed based on the fine-tuned visual encoder and the fine-tuned text encoder.

[0019] Step 3. In the first stage, the LoRA adapter implanted in the text encoder is set to a trainable state, and the fine-tuned visual-language model is used to extract features of the medical image and the text prompt set respectively to obtain visual representations and text features respectively. The weighted contrastive learning loss function of the first stage is constructed based on the visual representations and text features, and the text encoder is trained using the weighted contrastive learning loss function of the first stage to obtain the text encoder after the first stage training.

[0020] In step 3, in the first stage, the LoRA adapter embedded in the text encoder is set to a trainable state, and the fine-tuned visual-language model is used to extract features from the medical image and the text prompt set respectively to obtain visual representations and text features, respectively. A weighted contrastive learning loss function of the first stage is constructed based on the visual representations and text features, and the text encoder is trained using the weighted contrastive learning loss function of the first stage to obtain the text encoder after the first stage training. Specifically, the following sub-steps are included: In the first stage, the LoRA adapter embedded in the text encoder of the fine-tuned vision-language model is set to the trainable state and the visual encoder is frozen; The fine-tuned visual encoder is used to extract features from medical images to obtain a block feature sequence; The block feature sequence is input into the lightweight multi-scale visual feature fusion module for processing to obtain the visual representation. The following relationship exists in the corresponding process: ; in, represents visual representation, represents a lightweight multi-scale visual feature fusion module, represents the block feature sequence, represents the layer depth index of the visual encoder, Represents a predefined collection; It should be noted that the predefined set contains the numbers of all layers from which features are to be extracted. For example, if information of the 6th, 12th, 18th, and 24th layers is extracted from a 24-layer visual encoder, then the predefined set It is expressed as ,in this case, The values ​​will be 6, 12, 18 and 24 respectively.

[0021] Using a text encoder to perform feature extraction on the text prompt set to obtain text features; The weighted contrast loss function of the first stage is constructed based on visual representation and text features. The following relationship exists in the corresponding process: ; in, represents the weighted contrastive learning loss of the first stage, represents taking the logarithm, represents the exponential function, Indicates that after cosine similarity calculation, represents the positive sample text features corresponding to the medical image, represents the temperature hyperparameter, which is used to adjust the sharpness of the distribution; represents the key gradient weight factor, Indicates the a learnable text feature vector; Specifically, the calculation process of the key gradient weight factor has the following relationship: ; in, represents a hyperparameter.

[0022] The text encoder is trained using the weighted contrast learning loss function of the first stage to obtain the text encoder after the first stage training. The following relationship exists in the corresponding process: ; in, Indicates the optimal text LoRA parameters, It means taking the parameter that makes the function reach the minimum value. represents the trainable LoRA parameters in the text encoder, Represents the entire training dataset The average value of represents a medical image, represents a single "positive" text prompt, represents the visual encoder, represents the original parameters frozen in the visual encoder, represents a text encoder, Represents the frozen original parameters in the text encoder, Represents a single “negative” text prompt.

[0023] Furthermore, in the step of inputting the block feature sequence into the lightweight multi-scale visual feature fusion module for processing to obtain the visual representation, the following sub-steps are specifically included: The block feature sequences are spliced ​​in the sequence dimension to obtain the spliced ​​fusion feature vector; Performing a learnable position encoding configuration on the concatenated fusion feature vector to construct a position encoding matrix, and performing an embedding operation on the concatenated fusion feature vector using the position encoding matrix to obtain a fusion position encoding feature vector; The self-attention mechanism is used to process the feature vector of the fused position encoding to obtain the visual representation.

[0024] Step 4. In the second stage, the LoRA adapter implanted in the visual encoder is set to a trainable state, and the text encoder trained in the first stage is used to extract features of the text prompt set to obtain text anchor features. The weighted contrast learning loss function of the second stage is constructed based on the visual representation and the text anchor features. The visual encoder is trained using the weighted contrast learning loss function of the second stage to obtain the visual encoder trained in the second stage. The text encoder trained in the first stage and the visual encoder trained in the second stage constitute the trained visual-language model.

[0025] In step 4, in the second stage, the LoRA adapter implanted in the visual encoder is set to a trainable state, and the text encoder trained in the first stage is used to extract features of the text prompt set to obtain text anchor features. The weighted contrast learning loss function of the second stage is constructed based on the visual representation and the text anchor features. The visual encoder is trained using the weighted contrast learning loss function of the second stage to obtain the visual encoder trained in the second stage. The text encoder trained in the first stage and the visual encoder trained in the second stage are used to form a trained vision-language model. The specific sub-steps include the following: In the second stage, the LoRA adapter embedded in the text encoder after the first stage training is frozen, and the LoRA adapter embedded in the visual encoder of the fine-tuned vision-language model is set to the trainable state; Use the text encoder trained in the first stage to extract features from the text prompt set to obtain text anchor features; The weighted contrastive learning loss function of the second stage is constructed based on the visual representation and text anchor features. The following relationship exists in the corresponding process: ; in, represents the weighted contrastive learning loss of the second stage, represents the text anchor feature, Indicates the A fixed text feature vector; The visual encoder is trained using the weighted contrast learning loss function of the second stage to obtain the visual encoder after the second stage training. The following relationship exists in the corresponding process: ; in, represents the optimal visual LoRA parameters, Represents the trainable LoRA parameters in the visual encoder; The trained visual-language model is constructed using the text encoder trained in the first stage and the visual encoder trained in the second stage.

[0026] It's important to note that in the second phase, the text encoder and its LoRa adapter, trained in the first phase, are completely frozen. The original weight parameters of the visual encoder remain frozen, and only the embedded LoRa adapter is set to a trainable state. This is the core embodiment of "conditioning": visual fine-tuning is conditional on the final state of text fine-tuning. The core goal of the second phase is to train the visual encoder, guided by the solidified text anchor features, to learn how to extract visual features from the image that align with these precise semantics.

[0027] Step 5: Input the image to be tested, use the trained vision-language model to extract features from the image to be tested and the text prompt set, and construct classification and segmentation tasks respectively to obtain image-level anomaly scores and anomaly segmentation maps.

[0028] In step 5, the image to be tested is input, and the trained vision-language model is used to extract features from the image to be tested and the text prompt set. The classification task and segmentation task are then constructed to obtain an image-level anomaly score and anomaly segmentation map, respectively. The specific steps include the following: Input the image to be tested, and use the visual encoder trained in the second stage to extract features of the image to be tested to obtain enhanced visual features; The text encoder trained in the first stage is used to extract features from the text prompt set to obtain normal text anchor features and abnormal text anchor features respectively. The following relationship exists in the corresponding process: ; in, represents the normal text anchor feature, Indicates the total number of normal text prompts, Indicates the Normal text prompts, Represents abnormal text anchor features, Indicates the total number of abnormal text prompts. Indicates the Abnormal text prompts; Calculate the cosine similarity between the enhanced visual features, normal text anchor features, and abnormal text anchor features to obtain the image-level anomaly score. The following relationship exists in the corresponding process: ; in, represents the image-level anomaly score, Indicates enhanced visual features; The image-level anomaly score is judged by the classification threshold (the classification threshold can be adjusted according to the performance on the validation set). When the image-level anomaly score is greater than the classification threshold, the image is judged to be an abnormal sample; otherwise, it is judged to be a normal sample. The cosine similarity between the block feature sequence of each depth layer and the anchor feature of the abnormal text is calculated to generate an abnormal heat map. The following relationship exists in the corresponding process: ; in, Represents a heat map Middle The value of the position, Indicates that from The representative image extracted by the layer Block feature vector at each position; The abnormal heat maps of different depth layers are sequentially upsampled and weighted fused to obtain the fused total heat map. The following relationship exists in the corresponding process: ; in, represents the total heat map after fusion, Indicates the number of abnormal heat maps, Indicates that it has been upsampled; The fused total heat map is processed by Gaussian filtering to obtain the abnormal segmentation map. The following relationship exists in the corresponding process: ; in, represents the abnormal segmentation map, Represents a standard deviation of Gaussian kernel, Represents a convolution operation.

[0029] See also Figure 3 This embodiment further provides a unified anomaly detection system based on a conditional adapter, wherein the system applies the unified anomaly detection method based on a conditional adapter as described above, and the system includes: Prompt library building blocks for: Obtain domain-related seed questions and input them into a large language model for processing to obtain a text prompt set containing normal text prompts and abnormal text prompts; Adapter implant module for: Load the pre-trained vision-language model containing the visual encoder and text encoder, implant the LoRA adapter into the visual encoder and text encoder respectively, and freeze the original weight parameters of the visual encoder and text encoder to obtain the fine-tuned vision-language model; Two-stage training module for: In the first stage, the LoRA adapter embedded in the text encoder is set to a trainable state, and a fine-tuned vision-language model is used to extract features from medical images and text prompt sets to obtain visual representations and text features, respectively. A weighted contrastive learning loss function is constructed based on the visual representations and text features, and the text encoder is trained using the weighted contrastive learning loss function to obtain the text encoder after the first stage training. In the second stage, the LoRA adapter embedded in the visual encoder is set to a trainable state. The text encoder trained in the first stage is used to extract features from the text prompt set to obtain text anchor features. The weighted contrastive learning loss function of the second stage is constructed based on the visual representation and the text anchor features. The visual encoder is trained using the weighted contrastive learning loss function of the second stage to obtain the visual encoder trained in the second stage. The text encoder trained in the first stage and the visual encoder trained in the second stage are used to form a trained vision-language model. Anomaly detection module, used to: Input the image to be tested, use the trained vision-language model to extract features from the image to be tested and the text prompt set, and construct classification and segmentation tasks respectively to obtain image-level anomaly scores and anomaly segmentation maps respectively.

[0030] It should be understood that, although the various steps in the flow chart of each embodiment of the present invention are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0031] It should be understood that various components of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0032] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0033] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A unified anomaly detection method based on conditional adapter, characterized in that: The method comprises the following steps: Step 1: Obtain domain-related seed questions and input them into a large language model for processing to obtain a text prompt set containing normal text prompts and abnormal text prompts; Step 2: Load the pre-trained vision-language model including the visual encoder and the text encoder, implant the LoRA adapter into the visual encoder and the text encoder respectively, and freeze the original weight parameters of the visual encoder and the text encoder to obtain the fine-tuned vision-language model; Step 3. In the first stage, the LoRA adapter embedded in the text encoder is set to a trainable state, and the fine-tuned vision-language model is used to extract features from the medical image and the text prompt set to obtain visual representations and text features, respectively. A weighted contrastive learning loss function for the first stage is constructed based on the visual representations and text features, and the text encoder is trained using the weighted contrastive learning loss function for the first stage to obtain the text encoder after the first stage training. Step 4. In the second stage, the LoRA adapter implanted in the visual encoder is set to a trainable state, and the text encoder trained in the first stage is used to extract features of the text prompt set to obtain text anchor features. The weighted contrast learning loss function of the second stage is constructed based on the visual representation and the text anchor features. The visual encoder is trained using the weighted contrast learning loss function of the second stage to obtain the visual encoder trained in the second stage. The text encoder trained in the first stage and the visual encoder trained in the second stage are used to form a trained vision-language model. Step 5: Input the image to be tested, use the trained vision-language model to extract features from the image to be tested and the text prompt set, and construct classification and segmentation tasks respectively to obtain image-level anomaly scores and anomaly segmentation maps.

2. The unified anomaly detection method based on conditional adapter according to claim 1, characterized in that: In step 2, a pre-trained vision-language model including a visual encoder and a text encoder is loaded, the LoRA adapter is implanted into the visual encoder and the text encoder respectively, and the original weight parameters of the visual encoder and the text encoder are frozen to obtain a fine-tuned vision-language model, which specifically includes the following sub-steps: Load the pre-trained vision-language model containing the visual encoder and text encoder, implant the LoRA adapter into the visual encoder, and freeze the original weight parameters of the visual encoder to obtain the fine-tuned visual encoder; The LoRA adapter is implanted into the text encoder, and the original weight parameters of the text encoder are frozen to obtain a fine-tuned text encoder; A fine-tuned vision-language model is constructed based on the fine-tuned visual encoder and the fine-tuned text encoder.

3. The unified anomaly detection method based on conditional adapter according to claim 2, characterized in that: In step 3, in the first stage, the LoRA adapter embedded in the text encoder is set to a trainable state, and the fine-tuned visual-language model is used to extract features of the medical image and the text prompt set respectively to obtain visual representations and text features respectively. A weighted contrastive learning loss function of the first stage is constructed based on the visual representations and text features, and the text encoder is trained using the weighted contrastive learning loss function of the first stage to obtain a text encoder after training in the first stage. Specifically, the following sub-steps are included: In the first stage, the LoRA adapter embedded in the text encoder of the fine-tuned vision-language model is set to a trainable state, and the LoRA adapter embedded in the vision encoder is frozen; The fine-tuned visual encoder is used to extract features from medical images to obtain a block feature sequence; The block feature sequence is input into the lightweight multi-scale visual feature fusion module for processing to obtain visual representation; Perform feature extraction on the text prompt set using a fine-tuned text encoder to obtain text features; The weighted contrast loss function of the first stage is constructed based on visual representation and text features; The text encoder is trained using the weighted contrastive learning loss function of the first stage to obtain the text encoder after the first stage training.

4. The unified anomaly detection method based on conditional adapter according to claim 3, characterized in that: In the step of inputting the block feature sequence into the lightweight multi-scale visual feature fusion module for processing to obtain the visual representation, the following relationship exists: ； in, represents visual representation, represents a lightweight multi-scale visual feature fusion module, represents the block feature sequence, represents the layer depth index of the visual encoder, Represents a predefined collection; In the step of constructing the first-stage weighted contrast loss function based on visual representation and text features, the following relationship exists: ； in, represents the weighted contrastive learning loss of the first stage, represents taking the logarithm, represents the exponential function, Indicates that after cosine similarity calculation, Represents the positive sample text features corresponding to the medical image; represents the temperature hyperparameter, which is used to adjust the sharpness of the distribution; represents the key gradient weight factor, Indicates the a learnable text feature vector; In the step of training the text encoder using the weighted contrastive learning loss function of the first stage to obtain the text encoder after the first stage training, the following relationship exists: ； in, Indicates the optimal text LoRA parameters, It means taking the parameter that makes the function reach the minimum value. represents the trainable LoRA parameters in the text encoder, Represents the entire training dataset The average value of represents a medical image, represents a single "positive sample" text prompt, represents the visual encoder, represents the original parameters frozen in the visual encoder, represents a text encoder, Represents the frozen original parameters in the text encoder, Represents a single "negative" text prompt.

5. The unified anomaly detection method based on conditional adapter according to claim 4, characterized in that: The step of inputting the block feature sequence into the lightweight multi-scale visual feature fusion module for processing to obtain a visual representation specifically includes the following sub-steps: The block feature sequences are spliced ​​in the sequence dimension to obtain the spliced ​​fusion feature vector; The concatenated fused feature vector is configured with a learnable positional encoding to construct a positional encoding matrix, and the concatenated fused feature vector is embedded using the positional encoding matrix to obtain a fused positional encoding feature vector; The self-attention mechanism is used to process the feature vector of the fused position encoding to obtain the visual representation.

6. The unified anomaly detection method based on conditional adapter according to claim 5, characterized in that: In step 4, in the second stage, the LoRA adapter implanted in the visual encoder is set to a trainable state, and the text encoder trained in the first stage is used to extract features of the text prompt set to obtain text anchor features. A weighted contrast learning loss function of the second stage is constructed based on the visual representation and the text anchor features. The visual encoder is trained using the weighted contrast learning loss function of the second stage to obtain a visual encoder trained in the second stage. The text encoder trained in the first stage and the visual encoder trained in the second stage are used to form a trained visual-language model, which specifically includes the following sub-steps: In the second stage, the LoRA adapter embedded in the text encoder after the first stage training is frozen, and the LoRA adapter embedded in the visual encoder of the fine-tuned vision-language model is set to the trainable state; Use the text encoder trained in the first stage to extract features from the text prompt set to obtain text anchor features; The weighted contrastive learning loss function of the second stage is constructed based on the visual representation and text anchor features; The visual encoder is trained using the weighted contrast learning loss function of the second stage to obtain the visual encoder after the second stage training; The trained visual-language model is constructed using the text encoder trained in the first stage and the visual encoder trained in the second stage.

7. The unified anomaly detection method based on conditional adapter according to claim 6, characterized in that: In the step of constructing the second-stage weighted contrastive learning loss function based on the visual representation and text anchor features, the following relationship exists: ； in, represents the weighted contrastive learning loss of the second stage, represents the text anchor feature, Indicates the A fixed text feature vector; In the step of training the visual encoder using the weighted contrast learning loss function of the second stage to obtain the visual encoder after the second stage training, the following relationship exists: ； in, represents the optimal visual LoRA parameters, Represents the trainable LoRA parameters in the visual encoder.

8. The unified anomaly detection method based on conditional adapter according to claim 7, characterized in that: In step 5, the image to be tested is input, and the trained visual-language model is used to extract features from the image to be tested and the text prompt set. A classification task and a segmentation task are constructed respectively to obtain an image-level anomaly score and an anomaly segmentation map, respectively. The specific sub-steps include: Input the image to be tested, and use the visual encoder trained in the second stage to extract features of the image to be tested to obtain enhanced visual features; The text encoder trained in the first stage is used to extract features from the text prompt set to obtain normal text anchor features and abnormal text anchor features respectively; Calculate the cosine similarity between the enhanced visual features, normal text anchor features and abnormal text anchor features to obtain the image-level anomaly score; The image-level anomaly score is judged by the classification threshold. When the image-level anomaly score is greater than the classification threshold, the image to be tested is judged as an abnormal sample; otherwise, it is judged as a normal sample. Calculate the cosine similarity between the block feature sequence of each depth layer and the abnormal text anchor feature to generate an abnormal heat map; The abnormal heat maps of different depth layers are sequentially upsampled and weighted fused to obtain the fused total heat map; The fused total heat map is subjected to Gaussian filtering to obtain the abnormal segmentation map.

9. The unified anomaly detection method based on conditional adapter according to claim 8, characterized in that: In the step of extracting features from the text prompt set using the text encoder trained in the first stage to obtain normal text anchor features and abnormal text anchor features, the following relationship exists: ； in, represents the normal text anchor feature, Indicates the total number of normal text prompts, Indicates the Normal text prompts, Represents abnormal text anchor features, Indicates the total number of abnormal text prompts. Indicates the Abnormal text prompts; In the step of calculating the cosine similarity between the enhanced visual features, the normal text anchor features, and the abnormal text anchor features to obtain the image-level anomaly score, the following relationship exists: ； in, represents the image-level anomaly score, Indicates enhanced visual features; In the step of calculating the cosine similarity between the block feature sequence of each depth layer and the abnormal text anchor feature and generating the abnormal heat map, the following relationship exists: ； in, Representation heat map Middle The value of the position, Indicates that from The representative image extracted by the layer Block feature vector at each position; In the step of upsampling and weighted fusion of abnormal heat maps at different depth layers to obtain the fused total heat map, the following relationship exists: ； in, represents the total heat map after fusion, Indicates the number of abnormal heat maps, Indicates that it has been upsampled; In the step of performing Gaussian filtering on the fused total heat map to obtain the abnormal segmentation map, the following relationship exists: ； in, represents the abnormal segmentation map, Represents a standard deviation of Gaussian kernel, Represents a convolution operation.

10. A unified anomaly detection system based on conditional adapter, characterized in that: The system applies the unified anomaly detection method based on conditional adapter according to any one of claims 1 to 9, and the system includes: Prompt library building blocks for: Obtain domain-related seed questions and input them into a large language model for processing to obtain a text prompt set containing normal text prompts and abnormal text prompts; Adapter implant module for: Load the pre-trained vision-language model containing the visual encoder and text encoder, implant the LoRA adapter into the visual encoder and text encoder respectively, and freeze the original weight parameters of the visual encoder and text encoder to obtain the fine-tuned vision-language model; Two-stage training module for: In the first stage, the LoRA adapter embedded in the text encoder is set to a trainable state, and a fine-tuned vision-language model is used to extract features from medical images and text prompt sets to obtain visual representations and text features, respectively. A weighted contrastive learning loss function is constructed based on the visual representations and text features, and the text encoder is trained using the weighted contrastive learning loss function to obtain the text encoder after the first stage training. In the second stage, the LoRA adapter embedded in the visual encoder is set to a trainable state. The text encoder trained in the first stage is used to extract features from the text prompt set to obtain text anchor features. The weighted contrastive learning loss function of the second stage is constructed based on the visual representation and the text anchor features. The visual encoder is trained using the weighted contrastive learning loss function of the second stage to obtain the visual encoder trained in the second stage. The text encoder trained in the first stage and the visual encoder trained in the second stage are used to form a trained vision-language model. Anomaly detection module, used to: Input the image to be tested, use the trained vision-language model to extract features from the image to be tested and the text prompt set, and construct classification and segmentation tasks respectively to obtain image-level anomaly scores and anomaly segmentation maps respectively.

Images