Industrial image anomaly detection and segmentation method based on reinforcement learning and cross-modal alignment

By combining reinforcement learning and cross-modal alignment methods with format reward and focus reward mechanisms, high precision and efficiency of industrial image anomaly detection and segmentation are achieved, solving the problems of detection accuracy and annotation cost in existing technologies and providing an intelligent solution for industrial quality inspection.

CN120912993AActive Publication Date: 2025-11-07CHINA JILIANG UNIV +1

Patent Information

Application Number
CN202511430204.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-10-09
Publication Date
2025-11-07
Estimated Expiration
2045-10-09

AI Technical Summary

Technical Problem

Existing industrial image anomaly detection methods have limited generalization ability in complex scenes, lack cross-modal information utilization, and have low segmentation accuracy under weak supervision conditions and high sample annotation costs, making it difficult to meet the accuracy requirements of industrial detection.

Method used

We employ a reinforcement learning-based and cross-modal alignment approach, using the GRPO algorithm to fine-tune a large multimodal model, combined with format reward and focus reward mechanisms, to achieve hierarchical semantic fusion of image and text features, and utilize dynamic cue tokens for pixel-level segmentation.

Benefits of technology

It improves the accuracy of anomaly detection and the refinement of segmentation results under weak supervision, solves the problems of sample imbalance and high labeling costs, and provides an intelligent solution for industrial quality inspection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120912993A_ABST
    Figure CN120912993A_ABST
Patent Text Reader

Abstract

The invention discloses an industrial image anomaly detection and segmentation method based on reinforcement learning and cross-modal alignment. The method comprises the following specific steps: S1, establishing a multi-modal large model based on reinforcement learning fine tuning; s2, inputting an image to be queried and a prompt text into the trained multi-modal large model based on reinforcement learning fine tuning, outputting a reasoning text and judging whether an abnormal answer exists, and if the answer is abnormal, entering a step S3; s3, inputting an image to be queried and the reasoning text obtained in the step S2 into a cross-modal image-text alignment module to generate layered image-text features; and S4, inputting an image to be queried into an image segmentation module, mapping the image-text features obtained in the step S3 into a dynamic prompt token through a full connection layer, inputting the dynamic prompt token into a segmentation decoder, and generating a pixel-level abnormal segmentation result. According to the invention, the accuracy of anomaly detection and the refinement degree of the segmentation result are improved.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computers, and particularly relates to an industrial image anomaly detection and segmentation method based on reinforcement learning and cross-modal alignment. BACKGROUND

[0002] In the manufacturing and quality detection fields, image anomaly detection technology is crucial for ensuring product quality and production efficiency. However, existing methods still face many challenges in practical applications. Traditional anomaly detection is mainly based on manually designed features or supervised learning models, which not only require a large amount of labeled data, but also have limited generalization ability in complex industrial scenarios. Due to the scarcity of industrial defect samples and the high cost of labeling, model training is easily affected by the class imbalance problem, resulting in insufficient detection performance for abnormal samples. In addition, existing technologies are mostly limited to a single visual modality, lacking effective utilization of cross-modal information such as text prompts, making it difficult to achieve explainable anomaly reasoning.

[0003] In recent years, the development of multi-modal large models has provided new possibilities for combining visual and linguistic information, but their application in industrial scenarios still has room for optimization. For example, how to use reinforcement learning to fine-tune strategies to improve the model's ability to judge anomalies without masks, and how to achieve more accurate fusion of text and image features through cross-modal alignment, are problems that need to be solved. At the same time, in the absence of pixel-level labeling, existing weakly supervised segmentation methods often generate rough abnormal regions, making it difficult to meet the strict requirements of industrial detection for precision. Therefore, developing an anomaly detection method that can integrate multi-modal information, adapt to weak supervision conditions, and have high-precision segmentation capability is of great significance for promoting industrial intelligent detection. SUMMARY

[0004] To solve the above problems, the present application provides an industrial image anomaly detection and segmentation method based on reinforcement learning and cross-modal alignment, to solve the problems of high labeling cost of industrial defect samples, insufficient fusion of multi-modal information, and low segmentation precision under weak supervision conditions in the prior art.

[0005] The technical solution adopted by the present application is as follows: An industrial image anomaly detection and segmentation method based on reinforcement learning and cross-modal alignment, the specific steps of which are as follows: Step S1, establishing a multi-modal large model based on reinforcement learning fine-tuning; Step S2, inputting the image to be queried and the prompt text into the trained multi-modal large model based on reinforcement learning fine-tuning, outputting the reasoning text and the answer to whether there is an anomaly, and if the answer is that there is an anomaly, entering step S3; Step S3, input the image to be queried and the inference text obtained in step S2 into the cross-modal image-text alignment module to generate hierarchical image-text features; Step S4, input the image to be queried into the image segmentation module, and input the image-text features obtained in step S3 into the segmentation decoder through a fully connected layer to map the dynamic prompt token, to generate a pixel-level anomaly segmentation result.

[0006] Further, the training steps of the multi-modal large model based on reinforcement learning fine-tuning in step S1 are as follows: Step S1.1, obtain an industrial image dataset, and automatically generate corresponding prompt texts to form an image-text training set; Step S1.2, load a pre-trained multi-modal large model Qwen2-VL 7B; Step S1.3, input the image-text training set in step S1.1 into the multi-modal large model as training input, and fine-tune the multi-modal large model using the GRPO algorithm for reinforcement learning; Step S1.4, input the image-text for testing into the fine-tuned multi-modal large model, output the inference text and the answer to whether it is abnormal, and evaluate the performance of the multi-modal large model.

[0007] Further, the specific steps of step S1.3 for reinforcement learning fine-tuning include: Step S1.3.1, reward mechanism: use format reward to force the model to generate structured output, and use focus reward to guide the model to focus on difficult-to-classify samples and encourage the model to prioritize learning abnormal patterns; The expression of the format reward is as follows: ; The calculation method of the focus reward is as follows: ; Wherein, represents the prediction probability of the model for the correct class; represents an adjustment factor that controls the overall reward intensity; represents a focus parameter that improves the focus on low confidence samples; represents the importance weight of the key abnormal area, i.e., the proportion of the abnormal area; Step S1.3.2, advantage function normalization: standardize the rewards of each group of samples to calculate the advantage value of each sample : ; Wherein, represents the total reward of the current sample, respectively represent the mean and standard deviation of the current group of sample rewards, a small constant to prevent division by zero; Step S1.3.3, policy update: optimize the model based on the advantage function using the policy gradient method, and introduce the KL divergence term as regularization to keep the policy stable, the update formula is as follows: ; wherein, represents the current policy, represents the policy of the last step, represents the KL divergence between the policies, represents the learning rate, represents the adjustment coefficient of the KL term.

[0008] Further, the indicators for evaluating the performance of the model in step S1.4 include: Accuracy: measures the correctness of the overall prediction of the model, defined as follows: ; wherein, TP, TN, FP and FN represent true positive, true negative, false positive and false negative, respectively; Macro F1 score F1-macro: calculate the F1 score for each class separately and take the average, which pays more attention to the overall performance of each class in the case of class imbalance, defined as follows: ; wherein, N represents the number of classes, represents the F1 score of the i-th class, and the F1 score is the harmonic mean of precision and recall.

[0009] Further, the specific steps of the cross-modal image-text alignment module in step S3 to generate hierarchical image-text features are as follows: Step S3.1, after the image passes through the pre-trained CLIP image encoder, four levels of feature representations are extracted, representing different levels of image information: wherein, is the global classification feature of the image, is the local segmentation feature; Step S3.2, construct a learnable hierarchical text representation for the inference text, each layer of text is composed of a set of learnable embedded word vectors, containing semantic guide words, and the text features are also generated according to four levels, which are ; Step S3.3, use the image feature of each layer and the corresponding text feature to do hierarchical alignment: wherein, respectively represent normal and abnormal text features, and a cosine similarity is used to construct a hierarchical contrast learning target, h represents the i-th level, and t represents the text feature.

[0010] Further, step S3 further includes step S3.4 of weighting and fusing the matching results of each layer, and adaptively adjusting the contribution of each layer through a dynamic weight to obtain a final graph-text alignment score: ; wherein, is a learnable attention weight that dynamically adjusts the contribution of each layer.

[0011] Further, the image segmentation module in step S4 is implemented through weakly supervised training, and the specific steps are as follows: Step S4.1, a pre-trained Vision Transformer based image feature extraction is adopted, and image-text features are received simultaneously; Step S4.2, the image-text features are encoded into dynamic prompt tokens as input through a prompt encoder, and are input into a mask decoder together with image features to perform pixel-level segmentation inference of abnormal regions.

[0012] Further, the mask decoder in step S4.2 adopts a two-level Transformer decoding structure, and realizes the interaction between the image and the prompt feature through a cross-attention mechanism, and finally outputs a foreground probability map, which specifically includes: First attention: taking dynamic prompt tokens as queries, image embeddings as keys and values; Second attention: taking image embeddings as queries, and prompt tokens as keys and values; Normalization and residual: the output of each attention is normalized and connected through a residual connection before being passed to the next layer.

[0013] Further, the pixel-level segmentation inference of abnormal regions in step S4.2 adopts a multi-scale prediction mechanism, which is as follows: (1) 32x32 grid points are used to sample the initial candidate regions on the complete image, and 16x16 and 8x8 dense sampling are performed on 2x2 and 4x4 local cropped regions respectively, and after eliminating redundant proposals through non-maximum suppression, the highest confidence prediction result is selected; (2) After connected component filtering, hole filling and stability verification processing, an accurate pixel-level abnormal segmentation image is output.

[0014] Compared with the prior art, the present application has the following advantages: by combining reinforcement learning strategy with cross-modal feature alignment technology, and simultaneously fine-tuning the multi-modal large model by using the GRPO algorithm, a double incentive mechanism including format reward and focus reward is constructed, effectively solving the problems of sample imbalance and weakly supervised learning in industrial scenarios. In addition, the cross-modal image-text alignment module is used to realize the hierarchical semantic fusion of image and text features, and the weakly supervised segmentation mechanism based on dynamic prompt token breaks through the limitation of traditional methods in pixel-level positioning accuracy. This technical route of integrating reinforcement learning optimization, multi-level cross-modal understanding and adaptive segmentation simultaneously improves the accuracy of anomaly detection and the refinement degree of segmentation results under the condition of only relying on image-level weak labels, providing a new intelligent solution for industrial quality inspection. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is the overall framework flowchart of the present application.

[0016] Figure 2 is the cross-modal image-text alignment module structure diagram of the present application.

[0017] Figure 3 is the image segmentation module structure diagram of the present application.

[0018] Figure 4 is the structure diagram of the image segmentation module decoder of the present application. DETAILED DESCRIPTION

[0019] The present application will be further described below in conjunction with specific embodiments, but the present application is not limited to these specific embodiments. Those skilled in the art should realize that the present application encompasses all alternatives, improvements and equivalents within the scope of the claims.

[0020] Referring to Figures 1-4 , the embodiment provides an industrial image anomaly detection and segmentation method based on reinforcement learning and cross-modal alignment, and the specific steps are as follows: Step S1, establishing a multi-modal large model based on reinforcement learning (GRPO) fine-tuning;

[0021] The training steps of the multi-modal large model based on reinforcement learning fine-tuning are as follows: Step S1.1, obtaining an industrial image dataset, automatically generating corresponding prompt text to form an image-text training set; The industrial image dataset of the embodiment covers multiple industrial object categories and mixed complex scene categories. It contains 27 different image categories, including 15 common industrial products and 12 complex actual scenes. Among all the samples, the defect images account for about 17.7% of the total data, of which the defect sample proportion of the industrial object category is 23.8%, and the defect sample proportion of the complex scene category is 11.1%. All images come from the original dataset, only image-level labels (normal / defect) are used, and no pixel-level mask information is relied on. In addition, a corresponding text prompt is automatically generated for each image, and a unified prompt template is constructed combining the object name to guide the model to understand the image and judge the abnormality. Text prompt: example as follows: "This is a picture containing [object name]". Please analyze the picture and judge whether the object has defects.

[0022] Step S1.2, load the pre-trained multi-modal large model Qwen2-VL 7B; the embodiment adopts LoRA (Low-Rank Adaptation) and only fine-tunes part of the weights.

[0023] Step S1.3, input the image-text training set in step S1.1 into the multi-modal large model as training input, and use the GRPO algorithm to fine-tune the multi-modal large model by reinforcement learning; the GRPO (Group Relative Policy Optimization) is a reinforcement learning algorithm based on group normalization and KL divergence regularization, which is used to optimize the inference strategy of the model.

[0024] The specific steps of reinforcement learning fine-tuning include: Step S1.3.1, reward mechanism: use format reward to force the model to generate structured output, and use focus reward to guide the model to focus on difficult-to-classify samples and encourage the model to preferentially learn abnormal patterns; to guide the model to output structured and interpretable results, and to alleviate the common class imbalance problem in industrial image data.

[0025] The expression of format reward is as follows: ; The calculation method of focus reward is as follows: ; Wherein, represents the prediction probability of the model for the correct category; represents the adjustment factor, which controls the overall reward intensity; represents the focus parameter, which improves the attention to low confidence samples; represents the importance weight of the key abnormal area, i.e. the proportion of the abnormal area; Format reward and focus reward - respectively optimize the model from the dimensions of output specification and classification accuracy. Format reward forces the model to generate structured output (such as <think>and <answer>The label), ensuring that the results are resolvable and providing reliability for industrial deployment; the focus reward addresses the data imbalance problem by dynamically weighting (such as focusing on difficult samples and defect areas) to improve the sensitivity to rare defects. The connection between the two is that the format reward establishes the output framework, and the focus reward refines the content quality within this framework, and finally realizes "performance maximization under the rules" through the GRPO strategy.

[0026] Step S1.3.2, advantage function normalization: standardize the reward of each group of samples, and calculate the advantage value of each sample : ; wherein, represents the total reward of the current sample, respectively represent the mean and standard deviation of the current group of sample rewards, represents a small constant to prevent division by zero; The correct format is a prerequisite for the focus reward to take effect, but the value of the focus reward is not directly added to the format reward. The format reward (Format Reward) and the focus reward (Focal Reward) are combined and take effect according to conditions: (1) if the output format is incorrect → reward = 0 (format reward is 0, focus reward is not calculated); (2) if the format is correct → calculate the focus reward and take it as .

[0027] This step ensures that the model focuses on samples with relatively better or worse performance, effectively avoiding the problem of numerical instability during training.

[0028] Step S1.3.3, policy update: based on the advantage function, use the policy gradient method to optimize the model, and introduce the KL divergence term as regularization to maintain the stability of the policy, the update formula is as follows: ; wherein, represents the current policy, represents the policy of the previous step, represents the KL divergence between the policies, represents the learning rate, represents the adjustment coefficient of the KL term.

[0029] Through this updated policy, the model can continuously optimize its anomaly judgment ability and reasoning accuracy while ensuring the stability of the output.

[0030] Step S1.4, input the image-text for testing into the fine-tuned multi-modal large model, output the inference text and the answer of whether it is abnormal, and evaluate the performance of the multi-modal large model. Write the inference process in <analysis process>< / answer> < / think> In the tag, the final answer is written in the <answer> tag.

[0031] The indicators for evaluating the performance of the model in step S1.4 include: Accuracy: measures the correctness of the model's overall prediction, defined as follows: ; Where, TP, TN, FP and FN represent true positive, true negative, false positive and false negative, respectively; Macro F1 score F1-macro: take the average of F1 scores calculated for each class separately, pay more attention to the overall performance of each class in the case of class imbalance, defined as follows: ; Where N represents the number of classes, F1 represents the F1 score of the i-th class, and the F1 score is the harmonic mean of precision and recall.

[0032] Step S2, input the image to be queried and the prompt text into the trained multi-modal large model based on reinforcement learning fine-tuning, output the reasoning text and the answer whether there is an anomaly, if the answer is that there is an anomaly, go to step S3; Step S3, input the image to be queried and the reasoning text obtained in step S2 into the cross-modal image-text alignment module to generate hierarchical image-text features; The specific steps of the cross-modal image-text alignment module to generate hierarchical image-text features are as follows: Step S3.1, after the image passes through the pre-trained CLIP image encoder, four levels of feature representations (bottom, middle, high, semantic) are extracted, representing different levels of information of the image: Where, is the global classification feature of the image, is the local segmentation feature; Step S3.2, construct a learnable hierarchical text representation for the reasoning text to reduce the semantic difference between the image and the text at different levels. Each layer of text is composed of a set of learnable embedded word vectors, including semantic guide words such as [level], [good] representing normal, and [damage] representing abnormal. The text features are also generated according to four levels, respectively , as the input of the text encoder; Step S3.3, use the image features and the corresponding text features to do hierarchical alignment, where is the text encoder processed by step S3.2; ; wherein, respectively represent normal and abnormal text features, cosine similarity is used to construct the hierarchical contrast learning objective, h represents the i-th level, and t represents the text feature.

[0033] Step S3.4, the matching results of each layer are weighted and fused, and the dynamic weight is used to adaptively adjust the contribution of each layer to obtain the final image-text alignment score: ; wherein, is the learnable attention weight, which dynamically adjusts the contribution of each layer.

[0034] Hierarchical feature fusion: the alignment score is used to weight and fuse the image and text feature matching results of different levels. (2) Dynamic weight adjustment: through attention mechanism, the weight coefficients of each level are dynamically learned, so that the model can adaptively emphasize important levels according to task requirements (for example, industrial data may rely more on local details, while medical data needs global semantics).

[0035] Adjustment method: the weight coefficients are calculated through multi-layer perceptron and Softmax function. During training, these weights are optimized through backpropagation to minimize the total loss.

[0036] Step S4, input the image to be queried into the image segmentation module, and input the image-text features obtained in step S3 into the full connection layer to map them into dynamic prompt tokens, and input them into the segmentation decoder to generate pixel-level anomaly segmentation results.

[0037] The image segmentation module is realized through weakly supervised training, and the specific steps are as follows: Step S4.1, adopt pre-trained Vision Transformer (ViT) to extract image features, and simultaneously receive image-text features; Step S4.2, encode the image-text features into dynamic prompt tokens through the prompt encoder as input joint image features into the mask decoder for pixel-level segmentation inference of abnormal areas.

[0038] The mask decoder adopts a two-level Transformer decoding structure, which realizes the interaction between image and prompt features through cross-attention mechanism, and finally outputs the foreground probability map, which specifically includes: First layer attention: taking dynamic prompt token as query (Query), image embedding as key (Key) and value (Value); Second layer attention: image embedding as query, prompt token as key and value; Normalization and residual: each layer attention output is transmitted to the next layer after layer normalization (LayerNorm) and residual connection.

[0039] Among them, the pixel-level segmentation inference of the abnormal area adopts a multi-scale prediction mechanism, specifically as follows: (1) 32x32 grid points are sampled on the complete image to generate initial candidate regions, and 2x2 and 4x4 local cropped regions are densely sampled by 16x16 and 8x8 respectively. After eliminating redundant proposals by non-maximum suppression (threshold 0.7), the highest confidence prediction result is selected; (2) After connected component filtering (remove fragments with area <100 pixels), hole filling (fill gaps with area <100 pixels) and stability verification (double threshold 0.4 / 0.6 binary IoU >=0.95) processing, an accurate pixel-level abnormal segmentation image is output.

[0040] The application combines reinforcement learning strategy and cross-modal feature alignment technology, simultaneously fine-tunes multi-modal large model by using GRPO algorithm, constructs a double incentive mechanism containing format reward and focus reward, effectively solves the problems of sample imbalance and weak supervision learning in industrial scene. In addition, the cross-modal image-text alignment module realizes the hierarchical semantic fusion of image and text features, and the weak supervision segmentation mechanism based on dynamic prompt token breaks through the limitation of traditional method in pixel-level positioning accuracy. This technical route of fusion of reinforcement learning optimization, multi-level cross-modal understanding and adaptive segmentation improves the accuracy of anomaly detection and the refinement degree of segmentation results under the condition of only relying on image-level weak label, and provides a new intelligent solution for industrial quality inspection.

Claims

1. An industrial image anomaly detection and segmentation method based on reinforcement learning and cross-modal alignment, characterized in that: The specific steps are as follows: Step S1, establish a multi-modal large model based on reinforcement learning fine-tuning; Step S2, input the image to be queried and the prompt text into the trained multi-modal large model based on reinforcement learning fine-tuning, output the reasoning text and the answer of whether there is an anomaly, if the answer is an anomaly, enter step S3; Step S3, input the image to be queried and the reasoning text obtained in step S2 into the cross-modal image-text alignment module to generate hierarchical image-text features; Step S4, input the image to be queried to the image segmentation module, and input the image-text features obtained in step S3 to the full connection layer to map them into dynamic prompt tokens, and input them to the segmentation decoder to generate pixel-level anomaly segmentation results.

2. The industrial image anomaly detection and segmentation method based on reinforcement learning and cross-modal alignment according to claim 1, characterized in that: The training steps of the multi-modal large model based on reinforcement learning fine-tuning in step S1 are as follows: Step S1.1, obtain an industrial image dataset, and automatically generate corresponding prompt texts to form an image-text training set; Step S1.2, load a pre-trained multi-modal large model Qwen2-VL 7B; Step S1.3, input the image-text training set in step S1.1 into the multi-modal large model as training input, and use the GRPO algorithm to fine-tune the multi-modal large model by reinforcement learning; Step S1.4, input the image-text for testing into the fine-tuned multi-modal large model, output the reasoning text and the answer of whether there is an anomaly, and evaluate the performance of the multi-modal large model.

3. The industrial image anomaly detection and segmentation method based on reinforcement learning and cross-modal alignment according to claim 2, characterized in that: The specific steps of step S1.3 for reinforcement learning fine-tuning include: Step S1.3.1, reward mechanism: use format reward to force the model to generate structured output, and use focus reward to guide the model to focus on difficult classification samples and encourage the model to learn abnormal patterns preferentially; The expression of format reward is as follows: ; The calculation method of focus reward is as follows: ; wherein, represents the prediction probability of the model for the correct class; represents the adjustment factor, controlling the overall reward intensity; represents the focus parameter, improving the attention to low confidence samples; represents the importance weight of the key abnormal region, i.e. the abnormal region proportion; Step S1.3.2, advantage function normalization: standardize the rewards for each set of samples, compute the advantage value for each sample : ; wherein, represents the total reward of the current sample, respectively represent the mean and standard deviation of the current group sample reward, represents a small constant to prevent division by zero; Step S1.3.3, strategy update: optimize the model based on the advantage function using the policy gradient method, and introduce the KL divergence term as regularization to keep the strategy stable, the update formula is as follows: ; where, denotes the current policy, denotes the policy of the previous step, denotes the KL divergence between the policies, denotes the learning rate, denotes the adjustment coefficient of the KL term.

4. The industrial image anomaly detection and segmentation method based on reinforcement learning and cross-modal alignment according to claim 3, characterized in that: The performance evaluation indicators of the model in step S1.4 include: Accuracy: measures the correctness of the overall prediction of the model, defined as follows: ; wherein, true positive, true negative, false positive, and false negative, respectively; Macro F1 score F1-macro: calculates the F1 score for each class separately and takes the average, which is more concerned about the overall performance of each class in the case of class imbalance, defined as follows: ; where N represents the number of categories, F1 score of the i-th category, F1 score is the harmonic mean of precision and recall.

5. The industrial image anomaly detection and segmentation method based on reinforcement learning and cross-modal alignment according to claim 1, characterized in that: The specific steps of step S3 for the cross-modal image-text alignment module to generate hierarchical image-text features are as follows: Step S3.1, after the image passes through the pre-trained CLIP image encoder, four levels of feature representations are extracted, representing different levels of information of the image respectively: wherein, is the image global classification feature, is the local segmentation feature; Step S3.2, constructing a learnable hierarchical text representation from the inference text, each layer of text consisting of a set of learnable embedding word vectors, including semantic guide words, text features are also generated according to four levels, respectively ; Step S3.3, using image layer features and corresponding text features Do hierarchical alignment: ; where, represents normal and abnormal text features, respectively, and the cosine similarity is used to construct the hierarchical contrast learning objective, h represents the i-th level, and t represents the text feature.

6. The industrial image anomaly detection and segmentation method based on reinforcement learning and cross-modal alignment according to claim 5, characterized in that: Step S3 also includes step S3.4, which weights and fuses the matching results of each layer, and adjusts the contribution of each layer adaptively through dynamic weights to obtain the final image-text alignment score: ; wherein, are learnable attention weights that dynamically adjust the contribution of each layer.

7. The industrial image anomaly detection and segmentation method based on reinforcement learning and cross-modal alignment according to claim 1, characterized in that: The image segmentation module in step S4 is realized through weakly supervised training, and the specific steps are as follows: Step S4.1, use a pre-trained Vision Transformer to extract image features, while receiving image-text features; Step S4.2, encode the image-text features into dynamic prompt tokens as input through a prompt encoder, and input them to a mask decoder together with the image features for pixel-level anomaly segmentation reasoning.

8. The industrial image anomaly detection and segmentation method based on reinforcement learning and cross-modal alignment according to claim 7, characterized in that: The mask decoder in step S4.2 adopts a two-level Transformer decoding structure, realizes the interaction between the image and the prompt features through a cross-attention mechanism, and finally outputs a foreground probability map, which specifically includes: First layer attention: taking dynamic prompt tokens as queries and image embeddings as keys and values; Second layer attention: taking image embeddings as queries and prompt tokens as keys and values; Normalization and residual: the output of each layer of attention is transmitted to the next layer after layer normalization and residual connection.

9. The industrial image anomaly detection and segmentation method based on reinforcement learning and cross-modal alignment according to claim 7, characterized in that: The pixel-level segmentation inference of the abnormal area in step S4.2 adopts a multi-scale prediction mechanism, which is as follows: (1) On the complete image, 32x32 grid points are sampled to generate initial candidate regions, and 2x2 and 4x4 local cropped regions are respectively sampled densely at 16x16 and 8x8. After eliminating redundant proposals through non-maximum suppression, the highest confidence prediction result is selected; (2) After connected component filtering, hole filling and stability verification processing, an accurate pixel-level abnormal segmentation image is output.

Citation Information

Patent Citations

  • Heat supply system pipe network anomaly detection method based on multi-mode AI large model

    CN120430214A

  • Less-sample industrial anomaly detection method based on CLIP anomaly generation and text prompt

    CN120526219A

  • Lightweight CAN bus intrusion detection method based on enhanced active learning

    CN120614185A

  • Smart city security abnormal event detection method and system

    CN120673348A

  • Industrial zero sample anomaly detection method and system based on cross-modal prompt learning

    CN120726400A

Cited By

  • Industrial quality inspection method and system based on general quality inspection large model

    CN122067059A

  • Combined diagnosis method for self-regulation learning and emotion incentives

    CN122196643A

  • A method for joint diagnosis of self-regulated learning and emotional motivation

    CN122196643B