Image anomaly detection model training method, image anomaly detection method and device

Abstract

The embodiments of the present disclosure relate to the technical field of industrial defect detection, and specifically to an image anomaly detection model training method, an image anomaly detection method and a device. A specific implementation of the method disclosed herein includes: obtaining a training sample set, a normal category prompt text, an abnormal category prompt text, a learnable normal category prompt template, a learnable abnormal category prompt template and an initial image anomaly detection model; executing a model training step: inputting a normal category concatenated text vector and an abnormal category concatenated text vector into a text encoder to obtain normal category text features and abnormal category text features; performing image embedding processing on a sample image; inputting an image block embedding vector sequence into an image encoder to obtain a block feature sequence; inputting a block feature sequence, a normal category text feature and an abnormal category text feature into a segmentation decoder to obtain an abnormal segmentation prediction map; and determining an image anomaly detection model. This implementation can improve the efficiency of industrial product defect detection.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the technical field of industrial defect detection, and in particular to an image anomaly detection model training method, an image anomaly detection method and an image anomaly detection device. Background Art

[0002] Industrial anomaly detection aims to detect whether there are anomalies or defects in industrial products based on their images, and to accurately locate the defective areas from the images. At present, the common approach for industrial anomaly detection is to train an unsupervised anomaly detection model with normal samples to identify anomalies that deviate from the normal pattern during the inference phase, or to generate pseudo-abnormal samples or perform image restoration based on generative adversarial networks (GANs) or diffusion models.

[0003] However, it is found in practice that when the above method is adopted, the following technical problems often occur:

[0004] Since most unsupervised methods can only learn the distribution of normal samples, it is difficult to learn abnormal features when there are zero samples or scarce abnormal samples. In addition, industrial abnormal samples are usually scarce, and it is difficult to obtain sufficient labeled data during the production process. The generated pseudo-abnormal samples also have large feature differences from the samples in the real environment, which makes the recognition accuracy of the trained anomaly detection model low, and it is difficult to locate the defect area in a timely and accurate manner, thus resulting in low defect detection efficiency of industrial products.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose an image anomaly detection model training method, an image anomaly detection method and an apparatus to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide an image anomaly detection model training method, the method comprising: obtaining a training sample set, normal category prompt text, abnormal category prompt text, a learnable normal category prompt template, a learnable abnormal category prompt template and an initial image anomaly detection model, wherein each training sample in the above training sample set comprises a sample image and sample label data, and the initial image anomaly detection model comprises a text preprocessing module, an image preprocessing module, a text encoder, an image encoder and a segmentation decoder; selecting at least one training sample from the above training sample set, and performing the following model training steps based on the learnable normal category prompt template and the learnable abnormal category prompt template: performing splicing processing on the learnable normal category prompt template and the static normal category prompt text vector through the text preprocessing module included in the initial image anomaly detection model to obtain a normal category spliced ​​text vector, and performing splicing processing on the learnable abnormal category prompt template and the static abnormal category prompt text vector to obtain an abnormal category spliced ​​text vector, wherein the above static normal category prompt text vector is the result of pre-embedding the above normal category prompt text, and the above static abnormal category prompt text vector is the result of pre-embedding the above normal category prompt text. First, the result after text embedding of the above-mentioned abnormal category prompt text is obtained; the above-mentioned normal category spliced ​​text vector and the above-mentioned abnormal category spliced ​​text vector are input into the text encoder included in the initial image anomaly detection model to obtain normal category text features and abnormal category text features; through the image preprocessing module included in the initial image anomaly detection model, the sample image included in each training sample selected from at least one training sample is subjected to image embedding processing to generate an image block embedding vector sequence to obtain at least one image block embedding vector sequence; each image block embedding vector sequence in the above-mentioned at least one image block embedding vector sequence is input into the image encoder included in the initial image anomaly detection model to obtain at least one block feature sequence; the above-mentioned at least one block feature sequence, the above-mentioned normal category text features and the above-mentioned abnormal category text features are input into the segmentation decoder included in the initial image anomaly detection model to obtain at least one abnormal segmentation prediction map; based on the sample label data corresponding to each training sample in the above-mentioned at least one training sample and the corresponding abnormal segmentation prediction map, a training loss value is determined; in response to determining that the above-mentioned training loss value is less than the preset loss threshold, the trained initial image anomaly detection model is determined as the image anomaly detection model.

[0009] In a second aspect, some embodiments of the present disclosure provide an image anomaly detection method, the method comprising: acquiring an industrial product image; resizing the industrial product image to obtain an adjusted industrial product image; normalizing the pixel values ​​of the adjusted industrial product image to obtain a target image to be detected; inputting the target image to be detected into a pre-trained image anomaly detection model to obtain an anomaly segmentation prediction map corresponding to the target image to be detected, wherein the image anomaly detection model is trained by the method described in any implementation of the first aspect; in response to determining that the anomaly segmentation prediction map satisfies a preset anomaly pixel value condition, determining a preset image anomaly flag as the image detection result.

[0010] In a third aspect, some embodiments of the present disclosure provide an image anomaly detection model training device, the device comprising: an acquisition unit, configured to acquire a training sample set, a normal category prompt text, an abnormal category prompt text, a learnable normal category prompt template, a learnable abnormal category prompt template and an initial image anomaly detection model, wherein each training sample in the above-mentioned training sample set comprises a sample image and sample label data, and the initial image anomaly detection model comprises a text preprocessing module, an image preprocessing module, a text encoder, an image encoder and a segmentation decoder; a selection and execution unit, configured to select at least one training sample from the above-mentioned training sample set, and based on the learnable normal category prompt template and the learnable abnormal category prompt template, perform the following model training steps: splice the learnable normal category prompt template and the static normal category prompt text vector through the text preprocessing module included in the initial image anomaly detection model to obtain a normal category spliced ​​text vector, and splice the learnable abnormal category prompt template and the static abnormal category prompt text vector to obtain an abnormal category spliced ​​text vector, wherein the above-mentioned static normal category prompt text vector is the result of pre-embedding the above-mentioned normal category prompt text, and the above-mentioned static abnormal category prompt text vector is the result of pre-embedding the above-mentioned normal category prompt text, and the above-mentioned static abnormal category prompt text vector is the result of pre-embedding the above-mentioned normal category prompt text, and the above-mentioned static abnormal category prompt text vector is the result of pre-embedding the above-mentioned static abnormal category prompt text. The normal category prompt text vector is the result of pre-embedding the above-mentioned abnormal category prompt text; the above-mentioned normal category spliced ​​text vector and the above-mentioned abnormal category spliced ​​text vector are input into the text encoder included in the initial image anomaly detection model to obtain normal category text features and abnormal category text features; through the image preprocessing module included in the initial image anomaly detection model, the sample image included in each training sample of the selected at least one training sample is subjected to image embedding processing to generate an image block embedding vector sequence to obtain at least one image block embedding vector sequence; each image block embedding vector sequence in the above-mentioned at least one image block embedding vector sequence is input into the image encoder included in the initial image anomaly detection model to obtain at least one block feature sequence; the above-mentioned at least one block feature sequence, the above-mentioned normal category text features and the above-mentioned abnormal category text features are input into the segmentation decoder included in the initial image anomaly detection model to obtain at least one abnormal segmentation prediction map; based on the sample label data corresponding to each training sample in the above-mentioned at least one training sample and the corresponding abnormal segmentation prediction map, a training loss value is determined; in response to determining that the above-mentioned training loss value is less than the preset loss threshold, the trained initial image anomaly detection model is determined as the image anomaly detection model.

[0011] In a fourth aspect, some embodiments of the present disclosure provide an image anomaly detection device, comprising: an acquisition unit, configured to acquire an industrial product image; a resizing unit, configured to resize the industrial product image to obtain an adjusted industrial product image; a normalization processing unit, configured to perform pixel value normalization processing on the adjusted industrial product image to obtain a target image to be detected; an input unit, configured to input the target image to be detected into a pre-trained image anomaly detection model to obtain an anomaly segmentation prediction map corresponding to the target image to be detected, wherein the image anomaly detection model is trained by the method described in any implementation of the first aspect; a determination unit, configured to determine a preset image anomaly flag as the image detection result in response to determining that the abnormal segmentation prediction map satisfies a preset abnormal pixel value condition.

[0012] In a fifth aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first or second aspect above.

[0013] In a sixth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any implementation of the first aspect or the second aspect is implemented.

[0014] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the image anomaly detection model training method of some embodiments of the present disclosure, the defect detection efficiency of industrial products can be improved. Specifically, the reason for the low defect detection efficiency of industrial products is that most unsupervised methods can only learn the distribution of normal samples, and it is difficult to learn abnormal features when zero samples or abnormal samples are scarce. In addition, the number of industrial abnormal samples is usually scarce, and it is difficult to obtain sufficient labeled data during the production process. There are also large feature differences between the generated pseudo-abnormal samples and the samples of the real environment, which makes the recognition accuracy of the trained abnormal detection model low, and it is difficult to locate the defect area in a timely and accurate manner, thereby resulting in low defect detection efficiency of industrial products. Based on this, the image anomaly detection model training method of some embodiments of the present disclosure obtains a training sample set, a normal category prompt text, an abnormal category prompt text, a learnable normal category prompt template, a learnable abnormal category prompt template, and an initial image anomaly detection model. Among them, each training sample in the above-mentioned training sample set includes a sample image and a sample label data, and the initial image anomaly detection model includes a text preprocessing module, an image preprocessing module, a text encoder, an image encoder, and a segmentation decoder. Thus, data for training an image anomaly detection model can be obtained. Afterwards, at least one training sample is selected from the above training sample set, and based on the learnable normal category prompt template and the learnable abnormal category prompt template, the following model training steps are performed: first, through the text preprocessing module included in the initial image abnormality detection model, the learnable normal category prompt template and the static normal category prompt text vector are spliced ​​to obtain a normal category spliced ​​text vector, and the learnable abnormal category prompt template and the static abnormal category prompt text vector are spliced ​​to obtain an abnormal category spliced ​​text vector, wherein the static normal category prompt text vector is the result of pre-embedding the normal category prompt text, and the static abnormal category prompt text vector is the result of pre-embedding the abnormal category prompt text; the normal category spliced ​​text vector and the abnormal category spliced ​​text vector are input into the text encoder included in the initial image abnormality detection model to obtain normal category text features and abnormal category text features. Thus, text features with learnable category-independent prompt information can be obtained. Then, the image preprocessing module included in the initial image anomaly detection model performs image embedding processing on the sample image included in each training sample in the selected at least one training sample to generate an image block embedding vector sequence, and obtains at least one image block embedding vector sequence; each image block embedding vector sequence in the at least one image block embedding vector sequence is input into the image encoder included in the initial image anomaly detection model to obtain at least one block feature sequence. In this way, the encoded image features can be obtained.Next, the at least one block feature sequence, the normal category text features and the abnormal category text features are input into the segmentation decoder included in the initial image anomaly detection model to obtain at least one abnormal segmentation prediction map. Thus, an image characterizing the probability distribution of pixel anomalies for abnormal segmentation can be obtained. Finally, based on the sample label data corresponding to each training sample in the at least one training sample and the corresponding abnormal segmentation prediction map, a training loss value is determined; in response to determining that the training loss value is less than a preset loss threshold, the trained initial image anomaly detection model is determined as an image anomaly detection model. Therefore, the image anomaly detection model training method of some embodiments of the present disclosure, by adding category-independent text prompts to the input part of the text encoder and combining image features, can adapt to the anomaly detection of different objects across categories in the case of zero samples or scarce abnormal samples, improve the recognition accuracy of the model, and accurately locate the defective area. Thereby, the defect detection efficiency of industrial products can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0016] Figure 1 is a flowchart of some embodiments of the image anomaly detection model training method according to the present disclosure;

[0017] Figure 2 is a schematic diagram of a fixed text prompt as a classification surface according to some embodiments of the image anomaly detection model training method disclosed herein;

[0018] Figure 3 is a flow chart of some embodiments of the image anomaly detection method according to the present disclosure;

[0019] Figure 4 is an example effect diagram of abnormal segmentation according to some embodiments of the image abnormality detection method disclosed in the present invention;

[0020] Figure 5 is a schematic diagram of the structure of some embodiments of the image anomaly detection model training device according to the present disclosure;

[0021] Figure 6 is a schematic structural diagram of some embodiments of the image anomaly detection device according to the present disclosure;

[0022] Figure 7 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0024] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0025] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0026] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0028] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0029] Figure 1 The flowchart of some embodiments of the image anomaly detection model training method according to the present disclosure is shown. The image anomaly detection model training method 100 comprises the following steps:

[0030] Step 101, obtaining a training sample set, a normal category prompt text, an abnormal category prompt text, a learnable normal category prompt template, a learnable abnormal category prompt template and an initial image anomaly detection model.

[0031] In some embodiments, the execution subject (e.g., computing device) of the image anomaly detection model training method may obtain a training sample set, a normal category prompt text, an abnormal category prompt text, a learnable normal category prompt template, a learnable abnormal category prompt template, and an initial image anomaly detection model through a wired connection or a wireless connection. Each training sample in the training sample set may include a sample image and sample label data. The sample image may be an industrial image taken for an abnormality detection object for model training. The abnormality detection object may be an industrial product. For example, the abnormality detection object may be, but is not limited to, one of the following: metal parts, plastic products, and electronic components. For example, when the industrial product to be detected is a metal part, scratches or cracks on the surface of the metal part may be detected. The sample label data may include a sample classification label and a sample segmentation label. The sample classification label may represent the image category corresponding to the image. The image category may be one of the following: a normal category and an abnormal category. The normal category may represent that the object in the image does not have quality defects. The abnormal category may represent that the object in the image has quality defects. The sample classification label may be represented by 0 and 1. When the sample classification label is 0, the corresponding sample image is represented as an image of the normal category. When the sample classification label is 1, it indicates that the corresponding sample image is an image of an abnormal category. The sample segmentation label may be a true mask image of the sample image. The foreground portion of the true mask image is a pure white image area, which indicates a pre-marked abnormal area. The normal category prompt text may be a prompt word used to describe that the abnormal detection object is a normal category. The abnormal category prompt text may be a prompt word used to describe that the abnormal detection object is an abnormal category.

[0032] In addition, the above-mentioned learnable normal category prompt template may be a set of word vectors to be optimized for prompting images of normal categories. The above-mentioned learnable abnormal category prompt template may be a set of word vectors to be optimized for prompting images of normal categories. The initial image abnormality detection model may include a text preprocessing module, an image preprocessing module, a text encoder, an image encoder and a segmentation decoder. The above-mentioned text preprocessing module may be used to concatenate various text vectors. The above-mentioned image preprocessing module may be used to embed the input image. The above-mentioned text encoder may be used to encode text embedding features. The above-mentioned image encoder may be used to encode image embedding features. For example, the above-mentioned text encoder and the above-mentioned image encoder may be composed of CLIP (Contrastive Language-Image Pre-training). The above-mentioned segmentation decoder may be used to determine the probability that each pixel in the image is predicted to be a normal category or an abnormal category.

[0033] As an example, the execution subject may obtain a training sample set, normal category prompt text, abnormal category prompt text, learnable normal category prompt template and learnable abnormal category prompt template from a database, and obtain an initial image abnormality detection model from a model storage server. It should be noted that the wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0034] Optionally, before obtaining the training sample set, normal category prompt text, abnormal category prompt text, learnable normal category prompt template, learnable abnormal category prompt template and initial image anomaly detection model, the above-mentioned execution entity can also pre-train the above-mentioned text encoder and the above-mentioned image encoder.

[0035] Step 102, selecting at least one training sample from the training sample set, and performing the following model training steps based on the learnable normal category prompt template and the learnable abnormal category prompt template:

[0036] Step 1021, through the text preprocessing module included in the initial image anomaly detection model, the learnable normal category prompt template and the static normal category prompt text vector are spliced ​​to obtain the normal category spliced ​​text vector, and the learnable abnormal category prompt template and the static abnormal category prompt text vector are spliced ​​to obtain the abnormal category spliced ​​text vector.

[0037] In some embodiments, the execution subject may perform splicing processing on the learnable normal category prompt template and the static normal category prompt text vector through the text preprocessing module included in the initial image anomaly detection model to obtain a normal category splicing text vector, and perform splicing processing on the learnable abnormal category prompt template and the static abnormal category prompt text vector to obtain an abnormal category splicing text vector. The static normal category prompt text vector may be the result of pre-embedding the normal category prompt text. The static abnormal category prompt text vector may be the result of pre-embedding the abnormal category prompt text.

[0038] As an example, after obtaining the normal category prompt text and the abnormal category prompt text, the above-mentioned execution subject can embed the above-mentioned normal category prompt text and the above-mentioned abnormal category prompt text through a preset text embedding method to obtain a static normal category prompt text vector and the above-mentioned static abnormal category prompt text vector. Among them, the above-mentioned text embedding method can be a BERT (Bidirectional Encoder Representations from Transformers) model. Afterwards, the learnable normal category prompt template and the static normal category prompt text vector are spliced ​​in a horizontal splicing manner to obtain a normal category spliced ​​text vector, and the learnable abnormal category prompt template and the static abnormal category prompt text vector are spliced ​​to obtain an abnormal category spliced ​​text vector.

[0039] Optionally, the text encoder and the image encoder freeze parameters during the initial image anomaly detection model training process.

[0040] Step 1022: input the normal category concatenated text vector and the abnormal category concatenated text vector into a text encoder included in the initial image anomaly detection model to obtain normal category text features and abnormal category text features.

[0041] In some embodiments, the execution entity may input the normal category concatenated text vector and the abnormal category concatenated text vector into a text encoder included in the initial image abnormality detection model to obtain normal category text features and abnormal category text features. The normal category text features may be feature vectors representing normal category prompt texts. The abnormal category text features may be feature vectors representing abnormal category prompt texts.

[0042] Step 1023: Perform image embedding processing on the sample image included in each training sample of the at least one selected training sample through the image preprocessing module included in the initial image anomaly detection model to generate an image block embedding vector sequence, thereby obtaining at least one image block embedding vector sequence.

[0043] In some embodiments, the execution subject may perform image embedding processing on the sample image included in each training sample in the selected at least one training sample through the image preprocessing module included in the initial image anomaly detection model to generate an image block embedding vector sequence, and obtain at least one image block embedding vector sequence. Each image block embedding vector sequence in the at least one image block embedding vector sequence corresponds one-to-one to the sample image included in the training sample in the at least one training sample. The image block embedding vectors in the image block embedding vector sequence may represent image blocks in the same sample image.

[0044] As an example, the above-mentioned execution entity can, for each training sample in at least one training sample, perform average division processing on the sample image included in the above-mentioned training sample through the image preprocessing module included in the initial image anomaly detection model to obtain an image block sequence, and perform image embedding processing on each image block in the above-mentioned image block sequence to obtain an image block embedding vector sequence.

[0045] Step 1024: input each image block embedding vector sequence in at least one image block embedding vector sequence into an image encoder included in the initial image anomaly detection model to obtain at least one block feature sequence.

[0046] In some embodiments, the execution entity may input each image block embedding vector sequence in the at least one image block embedding vector sequence into an image encoder included in the initial image anomaly detection model to obtain at least one block feature sequence. Each block feature in the at least one block feature sequence corresponds one-to-one to an image block embedding vector in the at least one image block embedding vector sequence. The block features in the block feature sequence may be features of the corresponding image blocks. It should be noted that these block features contain local information of the sample image and retain the correspondence with the spatial layout of the original sample image.

[0047] Optionally, the image encoder may include an image coding layer sequence. The image coding layer in the image coding layer sequence may be a transformer coding layer. The input data of the target image coding layer may be composed of a pre-acquired learnable image hint vector, a learnable classification mark after encoding output by the previous image coding layer, and a block feature sequence after encoding. The learnable image hint vector may be a trainable vector used to guide or hint the model to better understand the image content. The previous image coding layer may be an image coding layer adjacent to the target image coding layer vector and executed earlier than the target image coding layer. The learnable classification mark after encoding may be a learnable classification mark after encoding processing. The learnable classification mark may be a trainable mark for classifying an image. The block feature sequence after encoding may be a block feature sequence obtained after encoding processing. The target image coding layer may be an image coding layer in the image coding layer sequence that meets a preset position condition. The preset position condition may be that the sequence number of the image coding layer is the same as the preset position sequence number. The input data of the non-target image coding layer may be the output data of the previous image coding layer. It should be noted that the input data of the first image coding layer in the image coding layer sequence may include an initial learnable classification tag and an image block embedding vector sequence. The initial learnable classification tag may be obtained by random initialization. Each image coding layer generates a corresponding encoded learnable classification tag and an encoded block feature sequence.

[0048] Step 1025: input at least one block feature sequence, normal category text features, and abnormal category text features into a segmentation decoder included in the initial image anomaly detection model to obtain at least one abnormal segmentation prediction map.

[0049] In some embodiments, the execution subject may input the at least one block feature sequence, the normal category text feature, and the abnormal category text feature into the segmentation decoder included in the initial image abnormality detection model in various ways to obtain at least one abnormal segmentation prediction map. Each abnormal segmentation prediction map in the at least one abnormal segmentation prediction map may correspond one-to-one to the sample image included in the training sample in the at least one training sample. Each abnormal segmentation prediction map in the at least one abnormal segmentation prediction map may represent the probability distribution of each pixel in the corresponding sample image being predicted as a normal category or an abnormal category.

[0050] In some optional implementations of some embodiments, the segmentation decoder may include a block feature alignment module and an upsampling module. The block feature alignment module may be used to align the image block features with the text features and determine the similarity between each block feature and the text feature. The upsampling module may be used to upsample the result of the block feature alignment module to obtain an abnormal segmentation prediction map with a resolution equivalent to that of the input sample image. The execution subject may input the at least one block feature sequence, the normal category text features and the abnormal category text features into the segmentation decoder included in the initial image anomaly detection model through the following steps to obtain at least one abnormal segmentation prediction map:

[0051] In the first step, the image-text similarity matrix corresponding to each block feature sequence in the at least one block feature sequence is determined based on the normal category text features and the abnormal category text features through the block feature alignment module included in the segmentation decoder to obtain at least one image-text similarity matrix. Each image-text similarity matrix in the at least one image-text similarity matrix corresponds one-to-one to the sample image included in the training sample in the at least one training sample. Each image-text similarity matrix in the at least one image-text similarity matrix can be a matrix with the block features of the corresponding sample image as rows, the normal category text features and the abnormal category text features as columns, and the image-text similarity between the block features and the text features as elements. The image-text similarity can characterize the degree of matching between the block features of the image and the normal category or the abnormal category.

[0052] As an example, the above-mentioned execution entity can determine the similarity between each block feature of the sample image and the normal category text feature and the abnormal category text feature according to the cosine similarity method for the sample image included in each training sample of the above-mentioned at least one training sample, through the block feature alignment module included in the above-mentioned segmentation decoder, to obtain an image-text similarity matrix.

[0053] In the second step, an upsampling module included in the segmentation decoder is used to upsample each image-text similarity matrix in the at least one image-text similarity matrix to generate an abnormal segmentation prediction map, thereby obtaining at least one abnormal segmentation prediction map.

[0054] As an example, the execution subject may perform upsampling on each of the at least one image-text similarity matrix by using an upsampling module included in the segmentation decoder according to a preset upsampling method to obtain an abnormal segmentation prediction map. The upsampling method may be, but is not limited to, one of the following: linear upsampling method, bilinear interpolation.

[0055] Step 1026: Determine a training loss value based on the sample label data corresponding to each training sample in at least one training sample and the corresponding abnormal segmentation prediction map.

[0056] In some embodiments, the execution subject may determine the training loss value in various ways based on the sample label data corresponding to each training sample in the at least one training sample and the corresponding abnormal segmentation prediction map. The training loss value may be generated by the classification loss and the segmentation loss. The classification loss may represent the difference between the prediction result when the model classifies the image and the sample classification label. The segmentation loss may represent the difference between the prediction result when the model segmented the image and the sample segmentation label.

[0057] It should be supplemented that, in the process of solving the above-mentioned technical problem 1 mentioned in the background technology part, the present disclosure is usually accompanied by the following technical problem 2: how to generate a training loss value to widen the difference between abnormal samples and normal samples. The conventional solution is usually to determine the training loss value of the model through the cross entropy loss function. However, the above-mentioned conventional solution still has the following problems: the cross entropy loss function can optimize the optimal threshold of all categories to 0.5, thereby improving the accuracy of the model, but when samples of different categories overlap in the feature space, these samples may have similar prediction probabilities, making it difficult for the cross entropy loss function to widen the distance between abnormal samples and normal samples, further making the performance of the image anomaly detection model low, and it is difficult to accurately detect defective products, thereby resulting in low defect detection efficiency of industrial products. Therefore, in the face of the above-mentioned technical problem 2 mentioned in the background technology part, combined with the technical advantages of the solution development team itself in the field of deep learning, the present disclosure decided to adopt the following solution.

[0058] In some optional implementations of some embodiments, the execution subject may determine the training loss value based on the sample label data and the corresponding abnormal segmentation prediction map corresponding to each training sample in the at least one training sample through the following steps:

[0059] In the first step, for each training sample in the at least one training sample, the following steps are performed:

[0060] The first sub-step is to generate an image prediction category probability based on the abnormal segmentation prediction map corresponding to the training sample. The image prediction category probability may be the maximum value among the pixel values ​​in the abnormal segmentation prediction map. The maximum value may be selected from the pixel values ​​included in the abnormal segmentation prediction map corresponding to the training sample as the image prediction category probability.

[0061] The second sub-step is to determine the coding layer prediction category probability corresponding to each intermediate layer in each intermediate layer based on the intermediate layer output results corresponding to the above training samples. Among them, each intermediate layer output result in the above intermediate layer output results can correspond to the intermediate layer in the preset intermediate layers one by one. Each intermediate layer output result in the above intermediate layer output results can be the output result of the corresponding intermediate layer. The intermediate layer in the above intermediate layers can be an image coding layer located at the intermediate level of the image coding layer sequence. For example, the image coding layer sequence in the image encoder includes 25 image coding layers, and each intermediate layer can be each image coding layer numbered 9, 15 and 21. For each intermediate layer output result in the intermediate layer output results corresponding to the above training samples, according to the intermediate layer output result and the above normal category text features and the above abnormal category text features, an abnormal segmentation prediction map of the corresponding intermediate layer is generated, and the maximum value of each pixel value included in the generated abnormal segmentation prediction map is determined as the coding layer prediction category probability of the corresponding intermediate layer.

[0062] The third sub-step is to generate a single sample loss value through a pre-designed loss function group based on the sample label data corresponding to the above training samples, the image prediction category probability, the output results of each intermediate layer, the determined prediction category probability of each coding layer, the preset classification weights of each intermediate layer and the segmentation weights of each intermediate layer. Among them, each intermediate layer classification weight and each intermediate layer segmentation weight can correspond one-to-one to the intermediate layers in the above intermediate layers. The intermediate layer classification weights in the above intermediate layer classification weights can be the preset weights of the corresponding intermediate layers in the image classification task. The intermediate layer segmentation weights in the above intermediate layer segmentation weights can be the preset weights of the corresponding intermediate layers in the image segmentation task. The above loss function group can be: .

[0063] in, Represents the single sample loss value. Represents the classification loss corresponding to the training sample. Represents the segmentation loss corresponding to the training sample. , Both represent the sequence number of the image coding layer. Represents a set of serial numbers corresponding to each intermediate layer. Indicates the sequence number of the last image coding layer sent in the image encoder. represents the classification loss function. Indicates The classification loss corresponding to the layer image encoding layer. Indicates The classification loss corresponding to the layer image encoding layer. represents the segmentation loss function. Indicates The segmentation loss corresponding to the layer image encoding layer. Indicates The segmentation loss corresponding to the layer image encoding layer. Indicates The intermediate layer classification weights corresponding to the layer image coding layer. Indicates The intermediate layer segmentation weights corresponding to the layer image coding layer. Indicates The classification loss corresponding to the layer image encoding layer. represents the cross entropy loss function. Indicates The cross entropy loss corresponding to the predicted output of the layer image coding layer. Represents the aggregation loss function. The above aggregation loss function can be used to maximize the interval between two different categories. Indicates The aggregation loss corresponding to the predicted output of the layer image coding layer. Indicates The segmentation loss corresponding to the predicted output of the layer image coding layer. represents the focal loss function. Indicates The focal loss corresponding to the predicted output of the layer image coding layer. Represents the Dice loss function. Indicates The Dice loss corresponding to the predicted output of the layer image coding layer. It should be noted that although and Included , but their scenarios and goals are different. It is image-level and mainly oriented to image classification tasks. It is pixel-level and mainly oriented towards image segmentation tasks.

[0064] As an example, the execution entity may determine the sum of the classification loss and the segmentation loss corresponding to the training sample as the single sample loss value.

[0065] Optionally, the above aggregation loss function can be designed as: .

[0066] in, Represents samples of the normal class. Represents samples of the abnormal class. Represents the classification threshold. In Indicates that the sample is predicted to be abnormal. Represents the anomaly score when a sample is predicted to be an anomaly. Represents the abnormality score of normal class samples. Represents the anomaly score of the abnormal class samples. Represents the corresponding aggregation loss when the sample belongs to the normal category. Indicates the corresponding aggregation loss when the sample is an abnormal category. Can be used to generate pixel-level aggregation loss, and at this time In represents a sample pixel of the normal category, and Sample pixels representing the anomaly class.

[0067] In practice, the above aggregation loss function can keep the anomaly scores of normal samples and abnormal samples at a length of The abnormal scores of normal samples and abnormal samples predicted incorrectly are clustered into and Nearby. As an example, It can be 0.5. Since semantic segmentation can be regarded as a pixel-level binary classification problem, the aggregation loss can also be extended to the pixel level.

[0068] Optionally, the above anomaly score It can be determined by the following formula group: .

[0069] in, Represents the result of image embedding of the sample image. Represents a collection of text vectors. Represents the normal category concatenated text vector. Represents the concatenated text vector of the abnormal category. Represents an image encoder. Represents a text encoder. Represents the similarity score between image and text. Indicates the preset temperature parameters. Represents the normalization function.

[0070] In practice, if the text encoder and text prompt are fixed, then Fixed, now is the image embedding feature The linear transformation of . Figure 2 , Figure 2 A schematic diagram of fixed text prompts as classification surfaces according to some embodiments of the image anomaly detection model training method disclosed herein. Figure 2 It includes three subgraphs (a), (b) and (c). and Projected into two-dimensional space, as shown in the following Figure 2 As shown in the sub-figure (a) of Figure 2, text hint anomaly detection essentially plays the role of classification in the binary classification problem. Figure 2 The (b) sub-figure in Fig. 2 shows that integrating textual hints into different classification aspects can reduce the bias of different classification aspects. Figure 2 The sub-figure (c) in FIG. 1 shows the classification surface of the image anomaly detection model according to the present disclosure that can be optimized during training, thereby obtaining a more generalized classification surface to be transferred to different data sets and different categories of images for classification.

[0071] In the second step, the sum of the obtained single sample loss values ​​is determined as the training loss value.

[0072] The above-mentioned single sample loss value generation step, as an inventive point of an embodiment of the present disclosure, solves the second technical problem mentioned in the background technology, "how to generate a training loss value to widen the difference between abnormal samples and normal samples". The reason why the conventional solution has a low efficiency in defect detection of industrial products is that the cross entropy loss function can optimize the optimal threshold of all categories to 0.5, thereby improving the accuracy of the model. However, when samples of different categories overlap in the feature space, these samples may have similar prediction probabilities, making it difficult for the cross entropy loss function to widen the distance between abnormal samples and normal samples, further making the performance of the image anomaly detection model low and difficult to classify defective products in time, thereby resulting in low defect detection efficiency of industrial products. If this solution solves the above-mentioned problems, it can achieve the effect of improving the defect detection efficiency of industrial products. In order to achieve this effect, the model loss of the present disclosure includes classification loss and segmentation loss. Among them, the classification loss of the sample is determined from the perspective of the image level, and the segmentation loss of the sample is determined from the perspective of the pixel level. Moreover, in order to make the anomaly score more linearly separable and the normal features and abnormal features more clustered, a clustering loss is designed, thereby widening the anomaly scores of normal samples and abnormal samples, and making the abnormal scores and normal scores of pixels have a significant gap. Furthermore, the abnormal area and the normal area can be significantly separable. In addition, by calculating the loss value of the intermediate level and introducing multi-level alignment, the accuracy of the model classification and segmentation results can be improved, thereby improving the performance of the image anomaly detection model and timely classifying defective products. In turn, the defect detection efficiency of industrial products can be improved.

[0073] Step 1027, in response to determining that the training loss value is less than a preset loss threshold, determining the trained initial image anomaly detection model as the image anomaly detection model.

[0074] In some embodiments, the execution subject may determine the trained initial image anomaly detection model as the image anomaly detection model in response to determining that the training loss value is less than a preset loss threshold, wherein the preset loss threshold may be an upper limit of the pre-set training loss value.

[0075] Optionally, the execution subject may also adjust the network parameters of the initial image anomaly detection model in response to determining that the training loss value is not less than the preset loss threshold, and use unused training samples to form a training sample set, and use the adjusted initial image anomaly detection model as the initial image anomaly detection model to perform the model training step again. The network parameters may include a learnable normal category prompt template, a learnable abnormal category prompt template, a learnable image prompt vector, and network parameters in a segmentation decoder. The network parameters of the initial image anomaly detection model may be adjusted by an AdamW optimizer and a cosine scheduler.

[0076] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the image anomaly detection model training method of some embodiments of the present disclosure, the defect detection efficiency of industrial products can be improved. Specifically, the reason for the low defect detection efficiency of industrial products is that most unsupervised methods can only learn the distribution of normal samples, and it is difficult to learn abnormal features when zero samples or abnormal samples are scarce. In addition, the number of industrial abnormal samples is usually scarce, and it is difficult to obtain sufficient labeled data during the production process. There are also large feature differences between the generated pseudo-abnormal samples and the samples of the real environment, which makes the recognition accuracy of the trained abnormal detection model low, and it is difficult to locate the defect area in a timely and accurate manner, thereby resulting in low defect detection efficiency of industrial products. Based on this, the image anomaly detection model training method of some embodiments of the present disclosure obtains a training sample set, a normal category prompt text, an abnormal category prompt text, a learnable normal category prompt template, a learnable abnormal category prompt template, and an initial image anomaly detection model. Among them, each training sample in the above-mentioned training sample set includes a sample image and a sample label data, and the initial image anomaly detection model includes a text preprocessing module, an image preprocessing module, a text encoder, an image encoder, and a segmentation decoder. Thus, data for training an image anomaly detection model can be obtained. Afterwards, at least one training sample is selected from the above training sample set, and based on the learnable normal category prompt template and the learnable abnormal category prompt template, the following model training steps are performed: first, through the text preprocessing module included in the initial image abnormality detection model, the learnable normal category prompt template and the static normal category prompt text vector are spliced ​​to obtain a normal category spliced ​​text vector, and the learnable abnormal category prompt template and the static abnormal category prompt text vector are spliced ​​to obtain an abnormal category spliced ​​text vector, wherein the static normal category prompt text vector is the result of pre-embedding the normal category prompt text, and the static abnormal category prompt text vector is the result of pre-embedding the abnormal category prompt text; the normal category spliced ​​text vector and the abnormal category spliced ​​text vector are input into the text encoder included in the initial image abnormality detection model to obtain normal category text features and abnormal category text features. Thus, text features with learnable category-independent prompt information can be obtained. Then, the image preprocessing module included in the initial image anomaly detection model performs image embedding processing on the sample image included in each training sample in the selected at least one training sample to generate an image block embedding vector sequence, and obtains at least one image block embedding vector sequence; each image block embedding vector sequence in the at least one image block embedding vector sequence is input into the image encoder included in the initial image anomaly detection model to obtain at least one block feature sequence. In this way, the encoded image features can be obtained.Next, the at least one block feature sequence, the normal category text features and the abnormal category text features are input into the segmentation decoder included in the initial image anomaly detection model to obtain at least one abnormal segmentation prediction map. Thus, an image characterizing the probability distribution of pixel anomalies for abnormal segmentation can be obtained. Finally, based on the sample label data corresponding to each training sample in the at least one training sample and the corresponding abnormal segmentation prediction map, a training loss value is determined; in response to determining that the training loss value is less than a preset loss threshold, the trained initial image anomaly detection model is determined as an image anomaly detection model. Therefore, the image anomaly detection model training method of some embodiments of the present disclosure, by adding category-independent text prompts to the input part of the text encoder and combining image features, can adapt to the anomaly detection of different objects across categories in the case of zero samples or scarce abnormal samples, improve the recognition accuracy of the model, and accurately locate the defective area. Thereby, the defect detection efficiency of industrial products can be improved.

[0077] Figure 3 The process of some embodiments of the image anomaly detection method according to the present disclosure is shown. The image anomaly detection method 300 comprises the following steps:

[0078] Step 301, acquiring an industrial product image.

[0079] In some embodiments, the execution subject (such as a computing device) of the image anomaly detection method can obtain industrial product images from a camera through a wired connection or a wireless connection. The industrial product images can be images taken by a camera of industrial products in the industrial field. It should be noted that the wireless connection method can include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultrawideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0080] Step 302: resize the industrial product image to obtain an adjusted industrial product image.

[0081] In some embodiments, the execution subject may resize the industrial product image to obtain an adjusted industrial product image. The adjusted industrial product image may be a scaled industrial product image. The industrial product image may be resized according to preset adjusted size information to obtain an adjusted industrial product image. The preset adjusted size information may be information about the width and height of the adjusted image that is preset. For example, the preset adjusted size information may be {"width": 336 pixels, "height": 336 pixels}.

[0082] Step 303 , performing pixel value normalization processing on the adjusted industrial product image to obtain a target image to be detected.

[0083] In some embodiments, the execution subject may perform pixel value normalization processing on the adjusted industrial product image to obtain the target image to be detected. The target image to be detected may be an adjusted industrial product image whose pixel values ​​obey a preset pixel distribution. The preset pixel distribution may be determined by a pre-generated pixel mean and pixel standard deviation. The pixel mean and the pixel standard deviation may be generated based on the same common image training data set. The pixel mean may be the global pixel mean of each image. The pixel standard deviation may be the global standard deviation of each image. The pixel value normalization processing may be performed on the adjusted industrial product image based on the pre-generated pixel mean and pixel standard deviation to obtain the target image to be detected.

[0084] Step 304: input the target image to be detected into a pre-trained image anomaly detection model to obtain an anomaly segmentation prediction map corresponding to the target image to be detected.

[0085] In some embodiments, the execution subject may input the target image to be detected into a pre-trained image anomaly detection model to obtain an anomaly segmentation prediction map corresponding to the target image to be detected. The image anomaly detection model may be pre-trained by the method of steps 101-1027.

[0086] Step 305: In response to determining that the abnormal segmentation prediction image satisfies a preset abnormal pixel value condition, a preset image abnormality flag is determined as the image detection result.

[0087] In some embodiments, the execution subject may determine a preset image abnormality flag as the image detection result in response to determining that the abnormal segmentation prediction map satisfies a preset abnormal pixel value condition. The preset abnormal pixel value condition may be that there is a pixel value greater than a preset pixel threshold in the abnormal segmentation prediction map. The preset pixel threshold may be an upper limit value of a preset pixel value. The preset image abnormality flag may characterize the industrial product image as an abnormal image. The abnormal image may be an image in which an industrial product in the image has defects.

[0088] Optionally, the execution subject may also determine a preset image no-abnormality mark as the image detection result in response to determining that the abnormal segmentation prediction map does not satisfy the preset abnormal pixel value condition. The preset image no-abnormality mark may indicate that the industrial product image is a no-abnormality image. The no-abnormality image may be an image in which the industrial product in the image has no defects.

[0089] Optionally, the above execution entity may further perform the following steps:

[0090] In the first step, in response to determining that the above-mentioned abnormal segmentation prediction map meets the preset abnormal pixel value condition, the abnormal segmentation prediction map corresponding to the target image to be detected is normalized to obtain a heat map corresponding to the target image to be detected. The normalization process may include normalizing pixel values, applying color mapping, and generating a heat map.

[0091] In the second step, the heat map corresponding to the target image to be detected is superimposed on the target image to be detected to obtain an abnormal segmentation image.

[0092] As an example, Figure 4 is an example effect diagram of abnormal segmentation according to some embodiments of the image abnormality detection method disclosed in the present invention. Figure 4 It includes 6 exemplary industrial images to be detected for anomalies, and also shows the segmented image after applying the real mask to each industrial product image and the predicted segmented image after applying the abnormal segmentation prediction map generated by the disclosed method. The predicted segmented image can be obtained by normalizing the abnormal segmentation prediction map into a heat map and overlaying it on the original industrial product image. The pure white area in the foreground of the real mask segmentation image is the pre-marked abnormal area. For the pixels on the heat map, the more the color is red, the higher the abnormality score, and the more it is blue, the lower the abnormality score, especially from Figure 4 It can be seen from the predicted segmentation image that the red abnormal area of ​​the heat map is very concentrated and has clear boundaries, and the normal areas of the background and the target are all in a uniform blue color. This shows that there is a significant gap between the abnormal score and the normal score of the image pixel points in the image anomaly detection method disclosed in the present invention, which helps to improve the effect of anomaly segmentation.

[0093] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: the image anomaly detection method of some embodiments of the present disclosure can improve the detection accuracy of industrial product images, timely discover defective industrial products, and improve the detection efficiency of defective products.

[0094] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an image anomaly detection model training device, and these device embodiments are Figure 1 Corresponding to the method embodiments shown, the image anomaly detection model training device 500 can be specifically applied to various electronic devices.

[0095] like Figure 5As shown, the image anomaly detection model training device 500 of some embodiments includes: an acquisition unit 501 and a selection and execution unit 502. Among them, the acquisition unit 501 is configured to acquire a training sample set, a normal category prompt text, an abnormal category prompt text, a learnable normal category prompt template, a learnable abnormal category prompt template and an initial image anomaly detection model, wherein each training sample in the above training sample set includes a sample image and sample label data, and the initial image anomaly detection model includes a text preprocessing module, an image preprocessing module, a text encoder, an image encoder and a segmentation decoder; the selection and execution unit 502 is configured to select at least one training sample from the above training sample set, and based on the learnable normal category prompt template and the learnable abnormal category prompt template, perform the following model training steps: through the text preprocessing module included in the initial image anomaly detection model, the learnable normal category prompt template and the static normal category prompt text vector are spliced ​​to obtain a normal category spliced ​​text vector, and the learnable abnormal category prompt template and the static abnormal category prompt text vector are spliced ​​to obtain an abnormal category spliced ​​text vector, wherein the above static normal category prompt text vector is the result of pre-embedding the above normal category prompt text, and the above static abnormal category prompt text vector is the result of pre-embedding the above normal category prompt text. The result after the abnormal category prompt text is embedded; the normal category concatenated text vector and the abnormal category concatenated text vector are input into the text encoder included in the initial image abnormality detection model to obtain normal category text features and abnormal category text features; the sample image included in each training sample in the selected at least one training sample is subjected to image embedding processing through the image preprocessing module included in the initial image abnormality detection model to generate an image block embedding vector sequence to obtain at least one image block embedding vector sequence; each image block embedding vector sequence in the at least one image block embedding vector sequence is input into the image encoder included in the initial image abnormality detection model to obtain at least one block feature sequence; the at least one block feature sequence, the normal category text features and the abnormal category text features are input into the segmentation decoder included in the initial image abnormality detection model to obtain at least one abnormal segmentation prediction map; based on the sample label data corresponding to each training sample in the at least one training sample and the corresponding abnormal segmentation prediction map, a training loss value is determined; in response to determining that the training loss value is less than the preset loss threshold, the trained initial image abnormality detection model is determined as the image abnormality detection model.

[0096] It can be understood that the units recorded in the image anomaly detection model training device 500 are similar to those in the reference Figure 1Therefore, the operations, features and beneficial effects described above for the method are also applicable to the image anomaly detection model training device 500 and the units contained therein, and will not be repeated here.

[0097] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an image anomaly detection device. These device embodiments are similar to Figure 3 Corresponding to the method embodiments shown, the image anomaly detection device 600 can be specifically applied to various electronic devices.

[0098] like Figure 6 As shown, the image anomaly detection device 600 of some embodiments includes: an acquisition unit 601, a resizing unit 602, a normalization processing unit 603, an input unit 604 and a determination unit 605. The acquisition unit 601 is configured to acquire an industrial product image; the resizing unit 602 is configured to resize the industrial product image to obtain the adjusted industrial product image; the normalization processing unit 603 is configured to perform pixel value normalization processing on the adjusted industrial product image to obtain the target image to be detected; the input unit 604 is configured to input the target image to be detected into a pre-trained image anomaly detection model to obtain an abnormal segmentation prediction map corresponding to the target image to be detected, wherein the image anomaly detection model is trained by the method described in steps 101-1027; the determination unit 605 is configured to determine the preset image anomaly identifier as the image detection result in response to determining that the abnormal segmentation prediction map meets the preset abnormal pixel value condition.

[0099] It can be understood that the units described in the image anomaly detection device 600 are similar to those described in the reference Figure 3 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the image anomaly detection device 600 and the units included therein, and will not be described in detail here.

[0100] Further references Figure 7 , which shows a structural schematic diagram of an electronic device 700 suitable for implementing some embodiments of the present disclosure. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0101] like Figure 7As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0102] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0103] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0104] It should be noted that the computer-readable medium mentioned above in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0105] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0106] The computer-readable medium may be included in the device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, as an example, the electronic device can execute: obtaining a training sample set, a normal category prompt text, an abnormal category prompt text, a learnable normal category prompt template, a learnable abnormal category prompt template and an initial image anomaly detection model, wherein each training sample in the training sample set includes a sample image and sample label data, and the initial image anomaly detection model includes a text preprocessing module, an image preprocessing module, a text encoder, an image encoder and a segmentation decoder; selecting at least one training sample from the training sample set, and performing the following model training steps based on the learnable normal category prompt template and the learnable abnormal category prompt template: splicing the learnable normal category prompt template and the static normal category prompt text vector through the text preprocessing module included in the initial image anomaly detection model to obtain a normal category spliced ​​text vector, and splicing the learnable abnormal category prompt template and the static abnormal category prompt text vector to obtain an abnormal category spliced ​​text vector, wherein the static normal category prompt text vector is the result of pre-embedding the normal category prompt text, and the static abnormal category prompt text vector is the result of pre-embedding the normal category prompt text. The category prompt text vector is the result of pre-embedding the above-mentioned abnormal category prompt text; the above-mentioned normal category splicing text vector and the above-mentioned abnormal category splicing text vector are input into the text encoder included in the initial image anomaly detection model to obtain normal category text features and abnormal category text features; through the image preprocessing module included in the initial image anomaly detection model, the sample image included in each training sample of the selected at least one training sample is subjected to image embedding processing to generate an image block embedding vector sequence to obtain at least one image block embedding vector sequence; each image block embedding vector sequence in the above-mentioned at least one image block embedding vector sequence is input into the image encoder included in the initial image anomaly detection model to obtain at least one block feature sequence; the above-mentioned at least one block feature sequence, the above-mentioned normal category text features and the above-mentioned abnormal category text features are input into the segmentation decoder included in the initial image anomaly detection model to obtain at least one abnormal segmentation prediction map; based on the sample label data corresponding to each training sample in the above-mentioned at least one training sample and the corresponding abnormal segmentation prediction map, a training loss value is determined; in response to determining that the above-mentioned training loss value is less than the preset loss threshold, the trained initial image anomaly detection model is determined as the image anomaly detection model.

[0107] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0108] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be arranged in a processor, for example, may be described as: a processor comprising: an acquisition unit and a selection and execution unit. Among them, the names of these units do not constitute a limitation on the unit itself in some cases, for example, the acquisition unit may also be described as "a unit for acquiring a training sample set, normal category prompt text, abnormal category prompt text, a learnable normal category prompt template, a learnable abnormal category prompt template and an initial image anomaly detection model".

[0110] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0111] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for training an image anomaly detection model, comprising: Acquire a training sample set, a normal category prompt text, an abnormal category prompt text, a learnable normal category prompt template, a learnable abnormal category prompt template, and an initial image anomaly detection model, wherein each training sample in the training sample set includes a sample image and sample label data, and the initial image anomaly detection model includes a text preprocessing module, an image preprocessing module, a text encoder, an image encoder, and a segmentation decoder; At least one training sample is selected from the training sample set, and based on the learnable normal category prompt template and the learnable abnormal category prompt template, the following model training steps are performed: Through the text preprocessing module included in the initial image anomaly detection model, the learnable normal category prompt template and the static normal category prompt text vector are horizontally spliced ​​to obtain a normal category spliced ​​text vector, and the learnable abnormal category prompt template and the static abnormal category prompt text vector are horizontally spliced ​​to obtain an abnormal category spliced ​​text vector, wherein the static normal category prompt text vector is the result of pre-embedding the normal category prompt text, the static abnormal category prompt text vector is the result of pre-embedding the abnormal category prompt text, the learnable normal category prompt template is to be optimized, composed of a group of word vectors for prompting images of normal categories, and the learnable abnormal category prompt template is to be optimized, composed of a group of word vectors for prompting images of abnormal categories; Inputting the normal category concatenated text vector and the abnormal category concatenated text vector into a text encoder included in the initial image abnormality detection model to obtain normal category text features and abnormal category text features, wherein the normal category text features are feature vectors representing normal category prompt texts, and the abnormal category text features are feature vectors representing abnormal category prompt texts; By using an image preprocessing module included in the initial image anomaly detection model, image embedding processing is performed on a sample image included in each training sample of the selected at least one training sample to generate an image block embedding vector sequence, thereby obtaining at least one image block embedding vector sequence; Inputting each image block embedding vector sequence in the at least one image block embedding vector sequence into an image encoder included in the initial image anomaly detection model to obtain at least one block feature sequence; Inputting the at least one block feature sequence, the normal category text feature and the abnormal category text feature into a segmentation decoder included in the initial image anomaly detection model to obtain at least one abnormal segmentation prediction map; Determine a training loss value based on the sample label data corresponding to each training sample in the at least one training sample and the corresponding abnormal segmentation prediction map; In response to determining that the training loss value is less than a preset loss threshold, the trained initial image anomaly detection model is determined as the image anomaly detection model.

2. The method according to claim 1, wherein: The method further comprises: In response to determining that the training loss value is not less than the preset loss threshold, adjusting the network parameters of the initial image anomaly detection model, and using unused training samples to form a training sample set, using the adjusted initial image anomaly detection model as the initial image anomaly detection model, and performing the model training step again.

3. The method according to claim 1, wherein: Before acquiring the training sample set, the normal category prompt text, the abnormal category prompt text, the learnable normal category prompt template, the learnable abnormal category prompt template and the initial image abnormality detection model, the method further includes: The text encoder and the image encoder are pre-trained.

4. The method according to claim 1, wherein: The text encoder and the image encoder freeze parameters during the initial image anomaly detection model training process.

5. The method according to claim 1, wherein: The image encoder includes an image coding layer sequence, the input data of the target image coding layer is composed of a pre-acquired learnable image prompt vector, a learnable classification mark output by a previous image coding layer, and a block feature sequence, the target image coding layer is an image coding layer in the image coding layer sequence that meets a preset position condition, and the input data of the non-target image coding layer is the output data of the previous image coding layer.

6. The method according to any one of claims 1 to 5, wherein: The segmentation decoder includes a block feature alignment module and an upsampling module; and the at least one block feature sequence, the normal category text feature and the abnormal category text feature are input into the segmentation decoder included in the initial image anomaly detection model to obtain at least one abnormal segmentation prediction map, including: By means of a block feature alignment module included in the segmentation decoder, based on the normal category text features and the abnormal category text features, an image-text similarity matrix corresponding to each block feature sequence in the at least one block feature sequence is determined to obtain at least one image-text similarity matrix; Through the upsampling module included in the segmentation decoder, each image-text similarity matrix in the at least one image-text similarity matrix is ​​upsampled to generate an abnormal segmentation prediction map, thereby obtaining at least one abnormal segmentation prediction map.

7. A method for detecting anomalies in an image, comprising: Acquire images of industrial products; resizing the industrial product image to obtain an adjusted industrial product image; Performing pixel value normalization processing on the adjusted industrial product image to obtain a target image to be detected; Inputting the target image to be detected into a pre-trained image anomaly detection model to obtain an anomaly segmentation prediction map corresponding to the target image to be detected, wherein the image anomaly detection model is trained by any one of the methods described in claims 1 to 6; In response to determining that the abnormal segmentation prediction image satisfies a preset abnormal pixel value condition, a preset image abnormality identifier is determined as the image detection result.

8. The method according to claim 7, wherein: The method further comprises: In response to determining that the abnormal segmentation prediction image does not satisfy the preset abnormal pixel value condition, a preset image no-abnormality flag is determined as the image detection result.

9. An image anomaly detection model training device, comprising: an acquisition unit configured to acquire a training sample set, a normal category prompt text, an abnormal category prompt text, a learnable normal category prompt template, a learnable abnormal category prompt template, and an initial image anomaly detection model, wherein each training sample in the training sample set includes a sample image and sample label data, and the initial image anomaly detection model includes a text preprocessing module, an image preprocessing module, a text encoder, an image encoder, and a segmentation decoder; The selection and execution unit is configured to select at least one training sample from the training sample set, and perform the following model training steps based on the learnable normal category prompt template and the learnable abnormal category prompt template: through the text preprocessing module included in the initial image abnormality detection model, the learnable normal category prompt template and the static normal category prompt text vector are horizontally spliced ​​to obtain the normal category spliced ​​text vector, and the learnable abnormal category prompt template and the static abnormal category prompt text vector are horizontally spliced ​​to obtain the abnormal category spliced ​​text vector, wherein the static normal category prompt text vector is the result of pre-text embedding the normal category prompt text, the static abnormal category prompt text vector is the result of pre-text embedding the abnormal category prompt text, the learnable normal category prompt template is to be optimized, composed of a group of word vectors for prompting normal category images, and the learnable abnormal category prompt template is to be optimized, composed of a group of word vectors for prompting abnormal category images; the normal category spliced ​​text vector and the abnormal category spliced ​​text vector are input into the initial image abnormality detection model, including A text encoder is provided to obtain normal category text features and abnormal category text features, wherein the normal category text features are feature vectors representing normal category prompt texts, and the abnormal category text features are feature vectors representing abnormal category prompt texts; through the image preprocessing module included in the initial image abnormality detection model, image embedding processing is performed on the sample image included in each training sample selected from at least one training sample to generate an image block embedding vector sequence, thereby obtaining at least one image block embedding vector sequence; each image block embedding vector sequence in the at least one image block embedding vector sequence is input into the image encoder included in the initial image abnormality detection model, thereby obtaining at least one block feature sequence; the at least one block feature sequence, the normal category text features, and the abnormal category text features are input into the segmentation decoder included in the initial image abnormality detection model, thereby obtaining at least one abnormal segmentation prediction map; a training loss value is determined based on the sample label data corresponding to each training sample in the at least one training sample and the corresponding abnormal segmentation prediction map; in response to determining that the training loss value is less than a preset loss threshold, the trained initial image abnormality detection model is determined as the image abnormality detection model.

10. An image anomaly detection device, comprising: An acquisition unit configured to acquire an industrial product image; a size adjustment unit, configured to adjust the size of the industrial product image to obtain an adjusted industrial product image; A normalization processing unit is configured to perform pixel value normalization processing on the adjusted industrial product image to obtain a target image to be detected; An input unit is configured to input the target image to be detected into a pre-trained image anomaly detection model to obtain an anomaly segmentation prediction map corresponding to the target image to be detected, wherein the image anomaly detection model is trained by any one of the methods described in claims 1 to 6; The determination unit is configured to determine a preset image abnormality flag as an image detection result in response to determining that the abnormal segmentation prediction image satisfies a preset abnormal pixel value condition.

Images