Interactive semi-automatic anomaly detection labeling method, system, equipment and medium

Abstract

The invention belongs to the technical field of computer vision, and discloses an interactive semi-automatic anomaly detection labeling method, system and device and a medium, and the method comprises the steps: obtaining a to-be-detected image of an industrial product, manual click input representing an abnormal region position, and natural language description of a defect type; based on the defect-free image reference set, extracting position constraint features of each image patch of the to-be-detected image, and calculating pixel-level position rapid abnormal residual features; converting the natural language description into a discriminant language feature vector by using a pre-training language model and a discriminant encoder; processing the residual feature through a residual feature branch, and generating a first fusion feature through a cross-modal attention mechanism under the guidance of a language feature vector; processing the original pixel information by combining the pixel information branch with click input to generate a second fusion feature; and fusing the two features and decoding to obtain a pixel-level abnormal annotation mask, thereby realizing high-precision semi-automatic annotation of the abnormal region in the industrial product image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to an image abnormal region labeling method for industrial product quality detection, and particularly to a semi-automatic abnormal detection labeling method, system, device and medium that fuses image features, language descriptions and manual interaction. BACKGROUND

[0002] In modern industrial manufacturing, product defect detection is a key link to ensure product quality. The anomaly detection (AD) technology based on machine vision, especially the one-class learning framework that only uses defect-free samples for training, has been widely applied. However, practice shows that introducing real defect samples into the training process can significantly improve the detection accuracy of the model. The key bottleneck of obtaining and utilizing these real defect samples lies in the need for accurate pixel-level labeling of the defect area, which traditionally relies on a large amount of manual operation, time-consuming and labor-intensive, and the labeling quality is affected by subjective factors, greatly limiting the large-scale deployment and application of advanced anomaly detection algorithms in manufacturing.

[0003] In order to solve the problem of lack of labeled data, there are some solutions in the prior art. One is supervised anomaly detection, which requires manual detailed labeling of all training data to clearly distinguish between normal and abnormal samples. Although this method is direct, its disadvantage is that the labeling cost is extremely high, which is almost impossible for large-scale data sets, and in many industrial scenarios, abnormal samples are very rare and it is difficult to collect enough samples for effective training.

[0004] Another solution is to generate defect samples through artificial manufacturing or algorithm simulation. For example, physical defects are manufactured on the product itself, or simulated defect patterns are superimposed on normal product images through image processing techniques. The disadvantage of this method is that artificially manufactured defects often differ in shape, texture and distribution from real defects, and it is difficult to fully reproduce the diversity and complexity of real-world defects, resulting in insufficient generalization ability of the model in real-world scenarios. At the same time, physical manufacturing of defects is costly and cumbersome.

[0005] Therefore, how to generate high-quality pixel-level labeling for a small number of real defect samples quickly and accurately with the lowest labor cost, so as to effectively improve the performance of the anomaly detection model, has become a technical problem to be solved in the field. SUMMARY

[0006] To address the aforementioned technical problems, this invention provides an interactive semi-automatic anomaly detection and annotation method, system, device, and medium. This method aims to achieve efficient and high-precision semi-automatic annotation of abnormal regions in industrial product images by fusing low-level visual features of images, high-level semantic language descriptions, and a small amount of precise manual interaction.

[0007] In a first aspect, the present invention provides an interactive semi-automatic anomaly detection and annotation method, comprising the following steps: Acquire an image of an industrial product to be inspected and at least one manual click input characterizing the location of an abnormal region in the image to be inspected; Based on a preset set of defect-free images, positional constraint features of each image patch in the image to be detected are extracted, and pixel-level positional fast anomaly residual features characterizing the difference between the image to be detected and the set of defect-free images are calculated. Natural language descriptions of defect types in the image to be detected are obtained, and the natural language descriptions are converted into discriminative language feature vectors using a pre-trained language model and a defect type discriminative encoder. A dual-branch feature processing network is used to process the location-based fast anomaly residual features and the pixel information of the image to be detected in parallel: In the first branch, namely the residual feature processing branch, the location fast anomaly residual features are processed using the first backbone network, and guided by the discriminative language feature vector, the location fast anomaly residual features are semantically weighted through a cross-modal attention mechanism to generate the first fusion feature. In the second branch, namely the pixel information processing branch, the original pixel information of the image to be detected is directly processed by using the second backbone network in conjunction with the manual click input to generate the second fusion feature; The first fusion feature and the second fusion feature are fused and decoded to generate a pixel-level anomaly label mask for the image to be detected.

[0008] As an optional implementation of the first aspect of this application, the step of calculating pixel-level fast anomaly residual features characterizing the difference between the image to be detected and the defect-free image reference set specifically includes: extracting a set of depth feature vectors from the image to be detected as a set of positional constraint features; extracting a set of anomaly-free reference features from the set of defect-free image references; and for each positional constraint feature in the set of positional constraint features, calculating the absolute value of the difference between it and the corresponding positional feature in the set of anomaly-free reference features to obtain the fast anomaly residual features.

[0009] As an optional implementation of the first aspect of this application, the step of converting the natural language description into a discriminative language feature vector specifically includes: using a large language model, based on preset template instructions, to automatically generate multiple candidate language descriptions for the defect type; inputting the candidate language descriptions into a pre-trained BERT model to generate initial language features; training the initial language features using a learnable defect type discriminative encoder, wherein the training process employs a joint loss function, the joint loss function including cross-entropy loss for defect type classification and L1 norm loss for feature reconstruction; the defect type classification explicitly divides defects into texture defects and structural defects.

[0010] As an optional implementation of the first aspect of this application, the step of semantically weighting the location fast anomaly residual features through a cross-modal attention mechanism specifically includes: using the location fast anomaly residual features processed by the first backbone network as a query vector Q; using the discriminative language feature vector as a key vector K and a value vector V; and using a scaled dot product attention mechanism to calculate the dot product attention score of the query vector Q, the key vector K, and the value vector V, thereby realizing the guidance and fusion of the discriminative language feature vector on the location fast anomaly residual features.

[0011] As an optional implementation of the first aspect of this application, the first backbone network is a SwingTransformer network, and the second backbone network is a Vision Transformer network; when the second branch combines the manual click input, it also fuses cross-modal features through a zero-convolution module to enhance the recognition of small-scale defects.

[0012] As an optional implementation of the first aspect of this application, the method further includes supporting iterative annotation optimization, specifically the following steps: after completing the generation of the pixel-level anomaly annotation mask once, the generated pixel-level anomaly annotation mask is used as the prior mask input for the next iteration; combined with additional manual click input provided by the user, the above steps of generating the pixel-level anomaly annotation mask of the image to be detected are repeatedly executed to gradually optimize and finally determine the pixel-level anomaly annotation mask.

[0013] As an optional implementation of the first aspect of this application, the method can also be applied to fully automated anomaly detection and localization tasks. The specific steps are as follows: without manual input, P possible defect types are preset, and a corresponding discriminative language feature vector is generated for each defect type; for each discriminative language feature vector corresponding to a defect type, the above steps of generating pixel-level anomaly labeling masks for the image to be detected are repeated to generate P candidate anomaly prediction maps; the P candidate anomaly prediction maps are pixel-level aggregated, and the maximum predicted value at each pixel position is taken to generate the final anomaly score map, so as to realize the detection and localization of unknown defect types.

[0014] Secondly, embodiments of this application provide an interactive semi-automatic anomaly detection and annotation system, including: An image acquisition module is used to acquire an image of an industrial product to be inspected and at least one manual click input characterizing the location of an abnormal region in the image to be inspected. The feature extraction module is used to extract the positional constraint features of each image patch in the image to be detected based on a preset defect-free image reference set, and to calculate the pixel-level positional fast anomaly residual features that characterize the difference between the image to be detected and the defect-free image reference set. The language description module is used to acquire a natural language description of the defect type in the image to be detected, and to convert the natural language description into a discriminative language feature vector using a pre-trained language model and a defect type discriminative encoder. The feature processing module is used to process the location fast anomaly residual features and the pixel information of the image to be detected in parallel through a dual-branch feature processing network: In the first branch, i.e., the residual feature processing branch, the location fast anomaly residual features are processed using the first backbone network, and guided by the discriminative language feature vector, the location fast anomaly residual features are semantically weighted through a cross-modal attention mechanism to generate a first fusion feature; In the second branch, i.e., the pixel information processing branch, the original pixel information of the image to be detected is directly processed using the second backbone network in conjunction with the manual click input to generate a second fusion feature; The feature fusion and decoding module is used to fuse the first fused feature and the second fused feature, and then decode them to generate a pixel-level anomaly label mask for the image to be detected.

[0015] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.

[0016] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Significantly reduce annotation costs: The interactive method of "few clicks + language description" replaces the traditional pixel-by-pixel manual drawing, which can reduce the time spent on manual annotation and greatly reduce labor and time costs.

[0018] 2. Improved annotation accuracy and consistency: By integrating pixel-level residual features, global semantic information, and precise user-defined pointers, the generated annotation mask has high accuracy, surpassing many traditional methods.

[0019] 3. Enhance model generalization ability: By introducing linguistic features that describe defect types, the model not only learns visual patterns but also understands the semantic concepts of defects, enabling it to better generalize to new and unseen defect types. It can quickly adapt by simply providing the corresponding linguistic description.

[0020] 4. High robustness of the method: The location-based rapid anomaly residual feature can effectively capture local and global anomalies in the image. The dual-branch structure complementarily processes residual information and original pixel information, enabling the method to have good detection performance for defects of various complex shapes and sizes (such as fine scratches, large-area discoloration, etc.).

[0021] 5. Strong applicability: This invention provides a complete lightweight and highly robust industrial quality inspection solution, which is not only suitable for semi-automatic labeling scenarios, but its fully automatic variant also provides a plug-and-play high-quality anomaly detection tool for automated production lines. Attached Figure Description

[0022] Figure 1 This is a flowchart of an interactive semi-automatic anomaly detection and annotation method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the network structure of the ADClick model proposed according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the algorithm flow for the language feature generation module proposed in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an interactive semi-automatic anomaly detection and labeling system provided in an embodiment of the present invention.

[0023] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0026] Example 1 Please see Figure 1 This invention provides an interactive semi-automatic anomaly detection and annotation method based on cross-modal feature fusion. For example... Figure 2 and Figure 3 As shown, the core idea of ​​this method is to take an image to be detected, several user clicks, and a defect description as input, and output a high-precision pixel-level anomaly mask through a carefully designed deep learning model (ADClick model).

[0027] The specific process of this method is as follows: S1. Acquire an image of the industrial product to be inspected and at least one manual click input characterizing the location of an abnormal region in the image to be inspected.

[0028] At the start of the method, the system acquires images of the industrial products to be processed. and a series of clicks made by the user on the image. ,in Indicates the height of the image. Indicates the width of the image. These are the pixel coordinates of the click. Indicates the first In each iteration, clicks are either negative samples (normal areas) or positive samples (defective areas). Simultaneously, the system receives natural language descriptions input by the user. For example, "the surface of the metal nut has scratches".

[0029] S2. Based on a preset defect-free image reference set, extract the positional constraint features of each image patch in the image to be detected, and calculate the pixel-level positional fast anomaly residual features that characterize the difference between the image to be detected and the defect-free image reference set.

[0030] To effectively capture the differences between the image to be detected and normal samples, this method uses the WeakREST algorithm to generate location fast anomaly residuals (PosFAR). The specific process is as follows: 1. Feature Extraction: A pre-trained deep network (such as WideResNet-50) is used as the feature extractor. The image to be detected... Extract a series of deep feature vectors This constitutes the set of position-constrained features (PCF). Similarly, for a pre-prepared reference set containing only defect-free images, its corresponding set of anomaly-free reference features is extracted. .

[0031] 2. Residual Calculation: For each image patch in the image to be detected, the positional constraint features are... Calculate its relationship with Corresponding position features Location rapid anomaly residual The calculation formula is as follows: in Represents the absolute value function. This indicates element-wise operation, where M represents the total number of feature patches into which the image to be detected is segmented. This represents the j-th position constraint feature in the image to be detected. The dimension of the feature vector. The residuals are features corresponding to the locations in the reference set, and θ is a hyperparameter used to adjust the sensitivity of the residuals. Compared to simple distance metrics, these residual features, which preserve intermediate matching information, have proven to be more robust. The residual features of all image patches are combined into a tensor set R.

[0032] S3. Obtain a natural language description of the defect type in the image to be detected, and use a pre-trained language model and defect type discriminant encoder to convert the natural language description into a discriminative language feature vector.

[0033] Use preset template instructions to control the automatic generation of ChatGPT models. Instead of using complex templates to generate language prompts, candidate language descriptions are selected. These candidate language descriptions are then input into a pre-trained BERT model to obtain the corresponding initial language features. The process of obtaining initial language features in the language prompt generation is as follows: Figure 3As shown, these cues have more variations, allowing the model to achieve higher generalization ability. Furthermore, the initial language features are trained using a learnable defect type discriminative encoder, employing a joint loss function that includes cross-entropy loss for defect type classification and L1 norm loss for feature reconstruction; the defect type classification explicitly categorizes defects into texture defects and structural defects.

[0034] Specifically, in order to describe natural language Transforming the features into discriminative feature vectors L that can be used for model computation, this method designs a language feature generation module: 1. Prompt Generation: To enhance the model's generalization ability and simulate diverse language expressions in the real world, a large number of candidate description statements are automatically generated using large language models (such as ChatGPT) and templated instructions (e.g., "Please describe the {defect type} on {object}").

[0035] 2. Feature Encoding and Training: These descriptive statements are input into a pre-trained BERT model to obtain initial language embeddings. To make these features more discriminative, this method designs a learnable encoder. This encoder is trained using a dual loss function.

[0036] During the training phase, enhanced cues are generated for all training images to train the language feature generation model. This method follows pioneering work by explicitly classifying descriptive statements into "texture" (local) and "structure" (global) categories based on the definition of defects. A learnable encoder is then trained to identify defect types. Specifically, given a set of training BERT features... And its defect type (texture or structure). encoder Learn through the following methods: in For joint losses; Cross-entropy loss is used to train the encoder to explicitly classify defect descriptions into two main categories: "texture" (such as stains, discoloration) or "structure" (such as cracks, missing parts). The L1 norm loss is used for feature reconstruction to ensure that the encoded features can be reconstructed back into the original BERT features by a decoder, thus preventing overfitting. in for Embedded linguistic features; To be generated by the encoder Obtained linguistic features Classifiers categorized into texture and structure classes. From A decoder that recovers BERT features; Small hyperparameters for balancing discriminative classification and feature reconstruction, used to balance the cross-entropy loss term and the L1 norm loss term; K represents the total number of BERT feature samples used to train the language feature encoder; Indicates the first k Defect description category labels corresponding to each BERT feature sample; This represents the feature vector of the k-th BERT feature sample after flattening.

[0037] After training in this way, the encoder can describe any defect in the input. Generate a highly condensed and discriminative (e.g., 512-dimensional) language feature vector L.

[0038] S4. Using a dual-branch feature processing network, the location fast anomaly residual features and the pixel information of the image to be detected are processed in parallel: In the first branch, i.e., the residual feature processing branch, the location fast anomaly residual features are processed using the first backbone network, and guided by the discriminative language feature vector, the location fast anomaly residual features are semantically weighted through a cross-modal attention mechanism to generate the first fusion feature; In the second branch, i.e., the pixel information processing branch, the original pixel information of the image to be detected is directly processed using the second backbone network in conjunction with the manual click input to generate the second fusion feature.

[0039] like Figure 2 As shown, the core of the ADClick model is a dual-branch network used to process visual residual information and raw pixel information in parallel.

[0040] 1. Residual Feature Processing Branch (First Branch): This branch receives the PosFAR tensor set R as input. To model long-range dependencies in the residual features, a Swing Transformer is used as the backbone network.

[0041] Cross-modal fusion: In the Swing Transformer module of this branch, a cross-attention mechanism is introduced. Specifically, the location-based fast anomaly residual features processed by the Swing Transformer are used as the query vector (Q) in the attention mechanism, while the generated discriminative language feature vector is used as L as the key vector (K) and value vector (V). By calculating the dot-product attention score of the query vector Q, key vector K, and value vector V through scaled dot-product attention, the language features can dynamically adjust the weights of different regions in the residual feature map, enabling the model to focus on matching anomaly patterns based on semantic information such as "scratches" and "holes." This process generates the first fused feature.

[0042] Specifically, R is used to generate query features, while the key and value vectors are obtained from the discriminative language feature vectors. Cross-modal fusion is achieved through scaled dot product attention, as shown below: in, Represents the query feature vector; This represents a combined function of a 1×1 convolutional layer and an instance normalization operation; This represents the height of the residual feature tensor R. The product of the residual feature tensor R and R represents the width of the residual feature tensor. That is, the total number of elements of the residual feature tensor after it is flattened in the spatial dimension; The key feature vector is generated by analyzing language features. l The result obtained by performing linear projection; This represents the learnable projection matrix used to generate key features; Indicates the feature used to generate values. The learnable projection matrix, and Similarly, its dimensional design needs to ensure that language features are included. l Linear mapping to dimension, The value feature vector is a linguistic feature. l via projection matrix The result obtained after linear transformation; The cross-modal fusion feature is the final fusion result obtained by scaling dot product attention.

[0043] 2. Pixel Information Processing Branch (Second Branch): This branch receives the original image I and the human click input C. It uses the Vision Transformer (ViT-B) as the backbone network to learn features directly from pixels. Click information is encoded in the form of distance maps and fused with image features. To enhance the ability to capture subtle defects, this branch also borrows ideas from ControlNet, introducing zero-convolutional modules to fuse cross-modal information. This process generates a second fused feature.

[0044] S5. Fuse the first fusion feature and the second fusion feature, and decode them to generate a pixel-level anomaly label mask for the image to be detected.

[0045] The first fused feature output from the residual branch and the second fused feature output from the pixel branch are concatenated or added and fed into a lightweight decoder module (such as several convolutional layers and upsampling layers). The decoder finally outputs a probability map of the same size as the input image, where the value of each pixel represents the probability that it belongs to an anomaly region. By setting a threshold, the final binarized pixel-level anomaly label mask M can be obtained.

[0046] Optionally, this method also includes step S6: iterative annotation optimization.

[0047] After generating the mask M1 for the first time, if the user finds the result imperfect, they can provide a new click C2. At this point, M1 will be used as a prior mask, inputting it along with the original image I and the new click C2 into the model for a second prediction. This process can be repeated until the user is satisfied with the annotation result. The previous prediction result provides valuable prior information for the next prediction, allowing subsequent corrections to typically be completed with only one or two clicks, greatly improving efficiency.

[0048] Specifically, in annotation iteration In the middle, the anomaly mask The prediction formula is as follows: Where the function This represents the ADClick model used in the Interactive Image Segmentation (IIS) algorithm. As the prediction mask from the previous iteration, we store the PosFAR features in a tensor. In, vector Linguistic features for task-guided statements that describe the types of defects in the input image.

[0049] Optionally, this method can also be applied to fully automated anomaly detection and localization tasks (ADClick-Seg).

[0050] In addition to the click model for interactive anomaly annotation, this method also designs a variant of the ADClick model, ADClick-Seg. In this mode, no manual clicks are required. Its workflow is as follows: 1. Simplified Model: Considering that automatic segmentation models are more prone to overfitting on smaller datasets compared to models with human intervention, the "fusion and decoder" module in ADClick-Seq has been significantly simplified, and the zero convolution module has been switched to the image branch.

[0051] 2. Full Defect Type Inference: Since the defect types are unknown during the testing phase, all P possible defect types are used for inference, such as scratches, stains, and damage. Corresponding linguistic feature vectors L1, L2, ..., L are generated for each type. P .

[0052] 3. Parallel Prediction and Aggregation: Combine the image to be detected I with each language feature vector L P The data are input into the ADClick-Seg model to obtain P candidate anomaly prediction maps. .

[0053] 4. Final score map generation: Perform pixel-level maximum aggregation on these P predicted maps. That is, the final anomaly score map. In coordinates The value at that location is: This final outlier score chart By combining the detection results of all possible defect types, it can effectively locate anomalies of unknown types.

[0054] Experimental results show that the ADClick model proposed in this invention can generate high-quality annotations with only 2 to 5 clicks on standard industrial inspection datasets such as MVTec AD, achieving an average precision (AP) of 96.1%. Its fully automatic variant, ADClick-Seg, also achieved a pixel-level AUC of 99.1% in the "one-for-all" task, demonstrating the advanced nature and effectiveness of the technical solution of this invention.

[0055] Example 2 Please see Figure 4 The diagram shown is a structural schematic of an interactive semi-automatic anomaly detection and annotation system proposed in the second embodiment of this application. The system includes the following key modules: Image acquisition module 100 is used to acquire an image of an industrial product to be inspected and at least one manual click input characterizing the location of an abnormal region in the image to be inspected. The feature extraction module 200 is used to extract the positional constraint features of each image patch in the image to be detected based on a preset defect-free image reference set, and to calculate the pixel-level positional fast anomaly residual features that characterize the difference between the image to be detected and the defect-free image reference set. The language description module 300 is used to acquire a natural language description of the defect type in the image to be detected, and to convert the natural language description into a discriminative language feature vector using a pre-trained language model and a defect type discriminative encoder. The feature processing module 400 is used to process the location fast anomaly residual features and the pixel information of the image to be detected in parallel through a dual-branch feature processing network: In the first branch, i.e., the residual feature processing branch, the location fast anomaly residual features are processed using the first backbone network, and guided by the discriminative language feature vector, the location fast anomaly residual features are semantically weighted through a cross-modal attention mechanism to generate a first fusion feature; In the second branch, i.e., the pixel information processing branch, the original pixel information of the image to be detected is directly processed using the second backbone network in conjunction with the manual click input to generate a second fusion feature; The feature fusion and decoding module 500 is used to fuse the first fused feature and the second fused feature, and decode them to generate a pixel-level anomaly label mask for the image to be detected.

[0056] An interactive semi-automatic anomaly detection and labeling system in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), etc. This application embodiment does not impose specific limitations.

[0057] The interactive semi-automatic anomaly detection and labeling system in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0058] The interactive semi-automatic anomaly detection and labeling system provided in this application embodiment can achieve...Figure 1 The various processes implemented by the interactive semi-automatic anomaly detection and annotation method in the method embodiment are not described in detail here to avoid repetition.

[0059] Optionally, embodiments of this application also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described interactive semi-automatic anomaly detection and annotation method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0060] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described interactive semi-automatic anomaly detection and annotation method and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0061] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0062] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0063] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0064] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An interactive semi-automatic anomaly detection and annotation method, characterized in that, Includes the following steps: Acquire an image of an industrial product to be inspected and at least one manual click input characterizing the location of an abnormal region in the image to be inspected; Based on a preset set of defect-free images, positional constraint features of each image patch in the image to be detected are extracted, and pixel-level positional fast anomaly residual features characterizing the difference between the image to be detected and the set of defect-free images are calculated. Natural language descriptions of defect types in the image to be detected are obtained, and the natural language descriptions are converted into discriminative language feature vectors using a pre-trained language model and a defect type discriminative encoder. A dual-branch feature processing network is used to process the location-based fast anomaly residual features and the pixel information of the image to be detected in parallel: In the first branch, namely the residual feature processing branch, the location fast anomaly residual features are processed using the first backbone network, and guided by the discriminative language feature vector, the location fast anomaly residual features are semantically weighted through a cross-modal attention mechanism to generate the first fusion feature. In the second branch, namely the pixel information processing branch, the original pixel information of the image to be detected is directly processed by using the second backbone network in conjunction with the manual click input to generate the second fusion feature; The first fusion feature and the second fusion feature are fused and decoded to generate a pixel-level anomaly label mask for the image to be detected.

2. The interactive semi-automatic anomaly detection and annotation method according to claim 1, characterized in that, The step of calculating the pixel-level location-based fast anomaly residual features characterizing the difference between the image to be detected and the defect-free image reference set specifically includes: Extract a set of depth feature vectors from the image to be detected as a set of positional constraint features; Extract an anomaly-free reference feature set from the defect-free image reference set; For each position constraint feature in the position constraint feature set, calculate the absolute value of the difference between it and the corresponding position feature in the non-anomaly reference feature set to obtain the position fast anomaly residual feature.

3. The interactive semi-automatic anomaly detection and annotation method according to claim 1, characterized in that, The step of converting the natural language description into a discriminative language feature vector specifically includes: Using a large language model and based on preset template instructions, multiple candidate language descriptions are automatically generated for each defect type. The candidate language descriptions are input into a pre-trained BERT model to generate initial language features; The initial language features are trained using a learnable defect type discrimination encoder. The training process employs a joint loss function, which includes cross-entropy loss for defect type classification and L1 norm loss for feature reconstruction. The defect type classification explicitly divides defects into texture defects and structural defects.

4. The interactive semi-automatic anomaly detection and annotation method according to claim 1, characterized in that, The step of semantically weighting the location-based fast anomaly residual features using a cross-modal attention mechanism specifically includes: The location fast anomaly residual features after processing by the first backbone network are used as the query vector Q; The discriminative language feature vectors are used as the key vector K and the value vector V; A scaling dot product attention mechanism is employed to calculate the dot product attention score of the query vector Q, key vector K, and value vector V, thereby enabling the discriminative language feature vector to guide and fuse the location fast anomaly residual features.

5. The interactive semi-automatic anomaly detection and annotation method according to claim 1, characterized in that, The first backbone network is a Swing Transformer network, and the second backbone network is a Vision Transformer network; The second branch, when combined with the manual click input, also fuses cross-modal features through a zero-convolution module to enhance the identification of small-scale defects.

6. The interactive semi-automatic anomaly detection and annotation method according to claim 1, characterized in that, The method also includes support for iterative annotation optimization, the specific steps of which are as follows: After generating the pixel-level anomaly label mask once, the generated pixel-level anomaly label mask is used as the prior mask input for the next iteration. By combining additional manual click input provided by the user, the steps described in claim 1 are repeated to progressively optimize and ultimately determine the pixel-level anomaly labeling mask.

7. An interactive semi-automatic anomaly detection and annotation method according to any one of claims 1 or 6, characterized in that, The method can also be applied to fully automated anomaly detection and localization tasks, with the following specific steps: Without human input, P possible defect types are preset, and a corresponding discriminative language feature vector is generated for each defect type. For each type of defect, the steps described in claim 1 are repeated to generate P candidate anomaly prediction maps. The P candidate anomaly prediction maps are aggregated at the pixel level, and the maximum predicted value at each pixel position is taken to generate the final anomaly score map, so as to realize the detection and location of unknown defect types.

8. An interactive semi-automatic anomaly detection and labeling system, characterized in that, include: An image acquisition module is used to acquire an image of an industrial product to be inspected and at least one manual click input characterizing the location of an abnormal region in the image to be inspected. The feature extraction module is used to extract the positional constraint features of each image patch in the image to be detected based on a preset defect-free image reference set, and to calculate the pixel-level positional fast anomaly residual features that characterize the difference between the image to be detected and the defect-free image reference set. The language description module is used to acquire a natural language description of the defect type in the image to be detected, and to convert the natural language description into a discriminative language feature vector using a pre-trained language model and a defect type discriminative encoder. The feature processing module is used to process the location fast anomaly residual features and the pixel information of the image to be detected in parallel through a dual-branch feature processing network: In the first branch, i.e., the residual feature processing branch, the location fast anomaly residual features are processed using the first backbone network, and guided by the discriminative language feature vector, the location fast anomaly residual features are semantically weighted through a cross-modal attention mechanism to generate a first fusion feature; In the second branch, i.e., the pixel information processing branch, the original pixel information of the image to be detected is directly processed using the second backbone network in conjunction with the manual click input to generate a second fusion feature; feature The fusion and decoding module is used to fuse the first fusion feature and the second fusion feature, and then decode them to generate a pixel-level anomaly label mask for the image to be detected.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of an interactive semi-automatic anomaly detection and annotation method as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps of an interactive semi-automatic anomaly detection and annotation method as described in any one of claims 1-7.

Images