An industrial defect image pixel-level labeling generation method and device

Abstract

The present application relates to the technical field of computer vision, and provides an industrial defect image pixel-level label generation method and device.The method comprises the following steps: extracting discriminative features of a defect image, using the discriminative features of the defect image to perform low-resolution pixel-level labeling on the defect image; extracting discriminative features of a normal image, using the discriminative features of the normal image to perform low-resolution pixel-level labeling on the normal image; fusing the discriminative features of the defect image and high-resolution features output by a VAE decoder to obtain refined features, and using the refined features to perform high-resolution pixel-level labeling on the defect image; and generating a target function according to the low-resolution pixel-level labeling and the high-resolution pixel-level labeling of the defect image and the normal image, a defect image mask and a normal image mask, and using the target function to optimize a labeling generation model.The present application can realize high-resolution pixel-level labeling that is accurately aligned with a defect area.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to an industrial defect image pixel-level label generation method and device. BACKGROUND

[0002] The existing methods for generating pixel-level labels of industrial defect images cannot meet the requirement of precise alignment of labels and defect parts in industrial scenarios. The method based on generative adversarial network usually requires hundreds of artificially labeled training images, which cannot meet the requirement of training a label generation model using only a small number of labeled defect images (generally within 5 images) in industrial scenarios. Some recent works based on diffusion models generate pixel-level labels by using cross-attention maps between text and images. However, due to the low resolution of cross-attention maps, up-sampling is required to obtain high-resolution pixel-level labels of the same resolution as the images. The up-sampling process causes the pixel-level labels of the defect region boundaries in the defect image to be unable to precisely align with the defect region, which affects the application effect of industrial defect images and labels in industrial defect detection tasks.

[0003] Therefore, it is urgent to overcome the defects of the prior art in the technical field. SUMMARY

[0004] The technical problem to be solved by the present application is that the prior art cannot achieve high-resolution pixel-level labels that are precisely aligned with the defect region.

[0005] The present application adopts the following technical solutions:

[0006] In a first aspect, the present application provides an industrial defect image pixel-level label generation method for training a label generation model and generating pixel-level labels using the trained label generation model. The label generation model includes a VAE encoder, a U-Net encoder, a U-Net decoder, a VAE decoder, a feature refinement module, and a mask prediction module. The label generation model is divided into a defect image branch and a normal image branch. The VAE encoder, the U-Net encoder, the U-Net decoder, the VAE decoder, the feature refinement module, and the mask prediction module constitute the defect image branch. The VAE encoder, the U-Net encoder, the U-Net decoder, the VAE decoder, and the mask prediction module also constitute the normal image branch. The training of the label generation model specifically includes:

[0007] The defect image branch extracts discriminative features of the defect image, and uses the discriminative features of the defect image to perform low-resolution pixel-level labeling on the defect image.

[0008] The normal image branch extracts discriminative features of the normal image, and uses the discriminative features of the normal image to perform low-resolution pixel-level labeling on the normal image.

[0009] The defect image branch fuses discriminative features of the defect image and high-resolution features output by the VAE decoder to obtain refined features, and uses the refined features to perform high-resolution pixel-level labeling on the defect image.

[0010] The low-resolution pixel-level labeling of the normal image is up-sampled to obtain high-resolution pixel-level labeling of the normal image.

[0011] According to the low-resolution pixel-level labeling of the defect image, the low-resolution pixel-level labeling of the normal image, the high-resolution pixel-level labeling of the defect image, the high-resolution pixel-level labeling of the normal image, the defect image mask and the normal image mask, a target function is generated, and the target function is used to optimize the labeling generation model.

[0012] Preferably, the discriminative features of the defect image include first features F1 and second features F2; the extraction of the discriminative features of the defect image and the low-resolution pixel-level labeling of the defect image using the discriminative features of the defect image specifically include:

[0013] The first features F1 output by the U-Net decoder are compressed in channel number using a 3x3 convolution block to obtain first compressed features F'1, and the second features F2 output by the U-Net decoder are compressed in channel number using a convolution block to obtain second compressed features F'2; wherein the first features F1 are features output by the second level of the U-Net decoder, and the second features F2 are features output by the third level of the U-Net decoder;

[0014] The first compressed features F'1 are up-sampled, and the result of the up-sampling of the first compressed features F'1 is spliced with the second compressed features F'2 to obtain spliced features F;

[0015] A plurality of transformation layers are applied to the spliced features F to obtain coarse features

[0016] A 3x3 convolution layer and a softmax function are applied to the coarse features to generate low-resolution pixel-level labeling of the defect image.

[0017] Preferably, the high-resolution features include an i-th high-resolution feature, i is an integer, i is greater than 0 and i is less than 4; the fusion of the discriminative features of the defect image and the high-resolution features output by the VAE decoder to obtain refined features specifically includes:

[0018] The i-th level feature is fused with the i-th high-resolution feature in sequence to obtain an (i+1)-th level feature, and the fourth level feature is taken as the refined feature; wherein, a discriminative feature of the defect image is taken as the first level feature;

[0019] The i-th level feature is fused with the i-th high-resolution feature in sequence to obtain an (i+1)-th level feature, and the fourth level feature is taken as the refined feature; wherein, a discriminative feature of the defect image is taken as the first level feature;

[0020] The i-th level feature is fused with the i-th high-resolution feature in sequence to obtain an (i+1)-th level feature, and the fourth level feature is taken as the refined feature; wherein, a discriminative feature of the defect image is taken as the first level feature;

[0021] The i-th level feature is fused with the i-th high-resolution feature in sequence to obtain an (i+1)-th level feature, and the fourth level feature is taken as the refined feature; wherein, a discriminative feature of the defect image is taken as the first level feature;

[0022] The i-th level feature is fused with the i-th high-resolution feature in sequence to obtain an (i+1)-th level feature, and the fourth level feature is taken as the refined feature; wherein, a discriminative feature of the defect image is taken as the first level feature;

[0023] The i-th level feature is fused with the i-th high-resolution feature in sequence to obtain an (i+1)-th level feature, and the fourth level feature is taken as the refined feature; wherein, a discriminative feature of the defect image is taken as the first level feature;

[0024] The i-th level feature is fused with the i-th high-resolution feature in sequence to obtain an (i+1)-th level feature, and the fourth level feature is taken as the refined feature; wherein, a discriminative feature of the defect image is taken as the first level feature;

[0025] The i-th level feature is fused with the i-th high-resolution feature in sequence to obtain an (i+1)-th level feature, and the fourth level feature is taken as the refined feature; wherein, a discriminative feature of the defect image is taken as the first level feature; The second channel of the defect pixel-level label is binarized using a preset threshold τ to obtain a high-resolution pixel-level label of the defect image.

[0026] The i-th level feature is fused with the i-th high-resolution feature in sequence to obtain an (i+1)-th level feature, and the fourth level feature is taken as the refined feature; wherein, a discriminative feature of the defect image is taken as the first level feature;

[0027] The i-th level feature is fused with the i-th high-resolution feature in sequence to obtain an (i+1)-th level feature, and the fourth level feature is taken as the refined feature; wherein, a discriminative feature of the defect image is taken as the first level feature;

[0028]

[0029] wherein, F() represents calculating Focal loss, and represent the low-resolution pixel-level label of the defect image and the high-resolution pixel-level label of the defect image respectively, and represent the low-resolution pixel-level label of the normal image and the high-resolution pixel-level label of the normal image respectively, and M df and M'df Let M represent the low-resolution ground truth mask and the high-resolution ground truth mask of the defect image, respectively. ob and M′ ob These represent the low-resolution true mask and the high-resolution true mask of a normal image, respectively.

[0030] Preferably, the defective image is pre-generated using a few-shot defective image generation model, which includes the U-Net encoder, the U-Net decoder, an attention module, a first objective function, a second objective function, and a third objective function; wherein the attention module, the first objective function, and the second objective function are assigned to the defective image branch, and the third objective function is assigned to the normal image branch; the training processes of the two branches are performed separately, but are implemented on a shared U-Net; a hybrid training strategy is used to train the few-shot defective image generation model, specifically including:

[0031] Multiple defective words and a single product word are combined to obtain an unbalanced anomaly text prompt. The unbalanced anomaly text prompt and the product-related part of the unbalanced anomaly text prompt are used as guiding conditions and input into the defective image branch and the normal image branch respectively.

[0032] A few sample defect images and their corresponding defect image masks are selected as inputs to the defect image branch. The defect image branch acquires the features of the defect images and aligns the defect words in the unbalanced abnormal text prompts to the defect regions in the images according to the defect image mask corresponding to the defect images, so as to optimize the first objective function and the second objective function.

[0033] A fixed number of normal images are selected as input to the normal image sample branch, which acquires the features of the normal images and uses the product-related parts of the imbalanced abnormal text prompts to optimize the third objective function; wherein, the hybrid training strategy is used to enable defect images of different defect categories to participate in training together.

[0034] Preferably, the first objective function is in, This is the cross-attention map corresponding to the nth defective word, where N is the number of defective words, L represents the number of U-Net layers corresponding to the cross-attention maps used in the alignment process, and M... l It is a binary mask, with a size of r×r pixels, where the value is 1 for defective areas and 0 for non-defective areas. This is the cross-attention map corresponding to product terms, where ⊙ represents multiplication by pixel;

[0035] The second objective function is Among them, edf z represents a conditional vector obtained after the unbalanced abnormal text prompt passes through a text encoder df x represents a defect image df ∈ represents a latent vector obtained through a VAE encoder df t represents random noise df n represents the number of noise addition steps z is obtained using ∈ df and t df z is obtained using ∈ df a noise-added latent vector obtained by adding noise to z z represents predicted noise.

[0036] Preferably, the third objective function is

[0037]

[0038] ; wherein e ob z represents a conditional vector obtained after the product-related part of the unbalanced abnormal text prompt passes through a text encoder ob x represents a normal image ob ∈ represents a latent vector obtained through a VAE encoder ob t represents random noise ob n represents the number of noise addition steps z is obtained using ∈ ob and t ob z is obtained using ∈ ob a noise-added latent vector obtained by adding noise to z z represents predicted noise.

[0039] In a second aspect, the present application further provides an industrial defect image pixel-level label generation device for implementing the industrial defect image pixel-level label generation method of the first aspect, and the device comprises:

[0040] at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the industrial defect image pixel-level label generation method of the first aspect.

[0041] In a third aspect, the present application further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method of the first aspect and / or the method of the second aspect.

[0042] In a fourth aspect, a chip is provided, comprising: a processor and an interface for calling and running a computer program stored in a memory to execute the method of the first aspect.

[0043] In a fifth aspect, there is provided a computer program product comprising instructions which, when executed on a computer or processor, cause the computer or processor to carry out the method of the first aspect.

[0044] The present application can realize high-resolution pixel-level labeling with accurate alignment to the defect area by introducing a VAE decoder to obtain high-resolution features of the defect image, fusing discriminative features and high-resolution features of the defect image to obtain refined features, and using the refined features to perform high-resolution pixel-level labeling on the defect image. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0046] Figure 1 is an architecture diagram of a labeling generation model in an industrial defect image pixel-level labeling generation method provided by the embodiments of the present application;

[0047] Figure 2 is a flowchart of a first industrial defect image pixel-level labeling generation method provided by the embodiments of the present application;

[0048] Figure 3 is a flowchart of a second industrial defect image pixel-level labeling generation method provided by the embodiments of the present application;

[0049] Figure 4 is a schematic diagram of an industrial defect image pixel-level labeling generation method provided by the embodiments of the present application;

[0050] Figure 5 is a flowchart of a third industrial defect image pixel-level labeling generation method provided by the embodiments of the present application;

[0051] Figure 6 is a flowchart of a fourth industrial defect image pixel-level labeling generation method provided by the embodiments of the present application;

[0052] Figure 7 is an architecture diagram of a few-sample defect image generation model in an industrial defect image pixel-level labeling generation method provided by the embodiments of the present application;

[0053] Figure 8 is a flowchart of a fifth industrial defect image pixel-level labeling generation method provided by the embodiments of the present application;

[0054] Figure 9 is a flowchart of a seventh industrial defect image pixel-level label generation method provided by an embodiment of the present application;

[0055] Figure 10 is a flowchart of a seventh industrial defect image pixel-level label generation method provided by an embodiment of the present application;

[0056] Figure 11 is a flowchart of a seventh industrial defect image pixel-level label generation method provided by an embodiment of the present application;

[0057] Figure 12 is a flowchart of a seventh industrial defect image pixel-level label generation method provided by an embodiment of the present application;

[0058] Figure 13 is a flowchart of a seventh industrial defect image pixel-level label generation method provided by an embodiment of the present application;

[0059] Figure 14 is a flowchart of a seventh industrial defect image pixel-level label generation method provided by an embodiment of the present application; DETAILED DESCRIPTION

[0060] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0061] Unless otherwise required by context, the term "comprises" in the specification and claims is to be construed as open-ended, i.e. as "comprises but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example" or "some examples" are intended to mean that the specific feature, structure, material or characteristic associated with that embodiment or example is included in at least one embodiment or example of the present disclosure. The illustrative representation of the above terms does not necessarily mean that the same embodiment or example is referred to. In addition, the specific features, structures, materials or characteristics described can be included in any one or more embodiments or examples in any appropriate manner, i.e. although they are carried in the embodiments or examples of the above terms due to the order of appearance and location, they are not limited to being carried by one embodiment or example in a combined manner.

[0062] In the description of the present application, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can be explicitly or implicitly included one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise stated, the meaning of "a plurality of" is two or more. In addition, for example, in the description, the same type of nouns will also be described as two independent individuals by adding "A", "B" at the end, in which case the features defined with "A", "B" are only used for the purpose of distinguishing the same type of individual description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features.

[0063] In the description of the present application, the expression "A and / or B" (where A and B represent specific feature content in form) includes the following three combinations: only A, only B, and the combination of A and B.

[0064] In the present application, "about", "approximately" or "approximately" includes the value stated and the average value within the acceptable deviation range of the specific value, wherein the acceptable deviation range is determined by considering the measurement being discussed and the error related to the measurement of the specific quantity (i.e. the limitation of the measurement system) by the person skilled in the art.

[0065] In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0066] Embodiment 1:

[0067] Embodiment 1 of the present application provides an industrial defect image pixel-level label generation method, a label generation model is trained, and a trained label generation model is used to generate pixel-level labels; as shown in Figure 1 The label generation model includes a VAE encoder, a U-Net encoder, a U-Net decoder, a VAE decoder, a feature refinement module and a mask prediction module; wherein it is also divided into a defect image branch and a normal image branch, the VAE encoder, the U-Net encoder, the U-Net decoder, the VAE decoder, the feature refinement module and the mask prediction module constitute the defect image branch, and the VAE encoder, the U-Net encoder, the U-Net decoder, the VAE decoder and the mask prediction module also constitute the normal image branch; the label generation model is trained, as shown in Figure 2 Specifically, it includes:

[0068] In step 201, the defect image branch extracts discriminative features of the defect image, and uses the discriminative features of the defect image to perform low-resolution pixel-level labeling on the defect image.

[0069] In step 202, the normal image branch extracts discriminative features of the normal image, and uses the discriminative features of the normal image to perform low-resolution pixel-level labeling on the normal image.

[0070] In step 203, the defect image branch fuses the discriminative features of the defect image and the high-resolution features output by the VAE decoder to obtain refined features, and uses the refined features to perform high-resolution pixel-level labeling on the defect image.

[0071] In step 204, the low-resolution pixel-level labeling of the normal image is up-sampled to obtain high-resolution pixel-level labeling of the normal image.

[0072] In step 205, a target function is generated according to the low-resolution pixel-level labeling of the defect image, the low-resolution pixel-level labeling of the normal image, the high-resolution pixel-level labeling of the defect image, the high-resolution pixel-level labeling of the normal image, the defect image mask and the normal image mask, and the labeling generation model is optimized using the target function.

[0073] The defect image can be a real image actually collected in an industrial scene. The low-resolution pixel-level labeling and the high-resolution pixel-level labeling are relative. In an actual application scenario, the low-resolution pixel-level labeling can refer to labeling with a pixel resolution of 64x64, and the high-resolution pixel-level labeling can refer to labeling with a pixel resolution of 512x512.

[0074] In the defect image branch, the defect image is input to the VAE encoder, the output result of the VAE encoder is input to the U-Net encoder, the output result of the U-Net encoder is input to the U-Net decoder, the output result of the U-Net decoder is input to the VAE decoder, and the VAE decoder processes the output result of the U-Net decoder to obtain the high-resolution features. The output result of the U-Net decoder is the discriminative features of the defect image.

[0075] In the normal image branch, the normal image is input to the VAE encoder, the output result of the VAE encoder is input to the U-Net encoder, the output result of the U-Net encoder is input to the U-Net decoder, and the U-Net decoder outputs the discriminative features of the normal image.

[0076] In the processing branch, the feature refinement module receives high-resolution features from the VAE decoder output and discriminative features of the defect image output by the U-Net decoder, processes to obtain refined features, and transmits the refined features to the mask prediction module. The defect image is also input into the mask prediction module. The mask prediction module uses the refined features to perform high-resolution pixel-level labeling on the defect image. The mask prediction module also receives discriminative features of the defect image output by the U-Net decoder and uses the discriminative features to perform low-resolution pixel-level labeling on the defect image. The mask prediction module also receives the normal image and discriminative features of the normal image output by the U-Net decoder, and performs low-resolution pixel-level labeling and high-resolution pixel-level labeling on the normal image according to the discriminative features of the normal image.

[0077] The process of optimizing the labeling generation model using the objective function can be understood as follows: through multiple iterations, the value of the objective function is calculated in each iteration, and the value of each parameter in the labeling generation model is adjusted according to the value of the objective function, so that the calculated value of each objective function tends to be ideal. The iteration process and the specific adjustment process of the parameters are known to those skilled in the art, and will not be described here.

[0078] In this embodiment, a VAE decoder is introduced to obtain high-resolution features of the defect image. Then, the discriminative features and the high-resolution features of the defect image are fused to obtain refined features, and the refined features are used to perform high-resolution pixel-level labeling on the defect image, so that high-resolution pixel-level labeling that is accurately aligned with the defect region can be achieved.

[0079] In actual application scenarios, the discriminative features of the defect image include first features F1 and second features F2. The discriminative features of the defect image are extracted, and the discriminative features of the defect image are used to perform low-resolution pixel-level labeling on the defect image, as shown in Figure 3 Specifically, the method comprises the following steps:

[0080] In step 301, a 3x3 convolution block is used to compress the channel number of the first feature F1 output by the U-Net decoder to obtain a first compressed feature F'1, and a convolution block is used to compress the channel number of the second feature F2 output by the U-Net decoder to obtain a second compressed feature F'2. The first feature F1 is the feature output by the second level of the U-Net decoder, and the second feature F2 is the feature output by the third level of the U-Net decoder. After compression, The second layer is also referred to as the "up-2" layer in practical use, and the third layer is also referred to as the "up-3" layer in practical use. The hierarchical structure of the U-Net decoder is as follows: Figure 4 As shown, it includes four levels: "up-1", "up-2", "up-3", and "up-4".

[0081] In step 302, the first compressed feature F′1 is upsampled, and the result of the upsampled first compressed feature F′1 is concatenated with the second compressed feature F′2 to obtain the concatenated feature F; the upsampling converts the first compressed feature F′1 to a resolution of 64×64.

[0082] In step 303, multiple transformation layers are applied to the spliced ​​feature F to obtain a coarse feature. Specifically, four transformation layers are applied to fuse the concatenated features F to obtain...

[0083] In step 304, in the rough feature A 3×3 convolutional layer and a softmax function are applied to generate low-resolution pixel-level annotations for the defect image. The low-resolution pixel-level annotations for the defect image are as follows:

[0084] The implementation method of using the discriminative features of the normal image to perform low-resolution pixel-level annotation of the normal image is based on the same concept as steps 301-304 above, and will not be described in detail here.

[0085] In one optional implementation, the high-resolution feature includes the i-th high-resolution feature, where i is an integer, greater than 0 and less than 4; the discriminative features of the defect image and the high-resolution features output by the VAE decoder are fused to obtain refined features, such as... Figure 5 As shown, it specifically includes:

[0086] The i-th level feature is fused with the i-th high-resolution feature in sequence to obtain the (i+1)-th level feature, and the fourth level feature is used as the refined feature; wherein, the discriminative feature of the defect image is used as the first level feature;

[0087] The process of fusing the i-th level feature with the i-th high-resolution feature to obtain the (i+1)-th level feature specifically includes:

[0088] In step 401, the i-th level feature is aligned with the i-th high-resolution feature.

[0089] In step 402, the i-th level feature is sequentially convolved with 1×1 and upsampled, and then multiplied element-wise with the corresponding aligned i-th high-resolution feature to obtain the i-th optimized boundary feature.

[0090] In step 403, the i-th level feature is added to the i-th optimized boundary feature after being processed by two convolution blocks to obtain the i+1-th level feature.

[0091] wherein the first high-resolution feature is the output of the first up-block of the VAE decoder, the second high-resolution feature is the output of the second up-block of the VAE decoder, and the third high-resolution feature is the output of the third up-block of the VAE decoder.

[0092] After each round of fusion, the obtained feature is more refined than the previous round. In this embodiment, multiple progressive fusions are performed in a cascading manner to gradually optimize the boundary of the coarse feature, and finally obtain a refined feature.

[0093] The refined feature is used for high-resolution pixel-level labeling of the defect image, as shown in Figure 6 The specific steps include:

[0094] In step 501, a 3x3 convolution layer and a softmax function are applied to the refined feature to generate a defect pixel-level label

[0095] In step 502, a preset threshold τ is used to binarize the second channel of the defect pixel-level label to obtain a high-resolution pixel-level label of the defect image. The preset threshold τ is obtained by demand analysis by those skilled in the art.

[0096] wherein the defect image mask includes a low-resolution real mask of the defect image and a high-resolution real mask of the defect image, and the normal image mask includes a low-resolution real mask of the normal image and a high-resolution real mask of the normal image; the target function is generated according to the low-resolution pixel-level label of the defect image, the low-resolution pixel-level label of the normal image, the high-resolution pixel-level label of the defect image, the high-resolution pixel-level label of the normal image, the defect image mask and the normal image mask, and specifically includes:

[0097] The target function is

[0098]

[0099] wherein F() represents the calculation of Focal loss, and represent the low-resolution pixel-level label of the defect image and the high-resolution pixel-level label of the defect image, respectively, and respectively represent low-resolution pixel-level labels of normal images and high-resolution pixel-level labels of normal images, M df and M' df respectively represent low-resolution real masks of defect images and high-resolution real masks of defect images, M ob and M' ob respectively represent low-resolution real masks of normal images and high-resolution real masks of normal images.

[0100] Embodiment 2:

[0101] At present, the generation methods in industrial scenes can be roughly divided into two categories: defect generation and defect image generation. The defect generation method simulates defects by pasting the cropped normal region to the normal image, and generates defects by different strategies and merges them into real normal images, or generates defects by using a diffusion model, and edits the defects to the normal image according to the defect mask guide.

[0102] However, the defect generation method relies on the defect mask as input, and if the mask position is unreasonable, the generated defect is easy to have low fidelity and poor consistency. In contrast, the defect image generation method can directly generate complete industrial defect images, which contain both industrial defects and industrial products with industrial defects. This kind of method avoids the dependence on the mask, so it can generate more natural and diverse defect instances, but the defect image generation method needs to use a large number of defect images to train the diffusion model in advance. Few-sample industrial defect image generation refers to the training of an industrial defect image generation model based on a small number of training images (generally within 5 images). For the same industrial product, there are usually defect images of different defect categories. Since the occurrence of defects in the industrial scene is a small probability event under normal circumstances, it is difficult to quickly collect a large number of defect images for each defect, so few samples are common in industrial scenes. The sample size for training is small, making it difficult to achieve ideal training results. In order to solve this problem, the present embodiment further provides an industrial defect image pixel-level label generation method on the basis of Embodiment 1, and the defect image used in the industrial defect image pixel-level label generation method is generated by using a few-sample defect image generation model, as shown in Figure 7 The few-sample defect image generation model includes the U-Net encoder of Embodiment 1, the U-Net decoder of Embodiment 1, an attention module, a first target function, a second target function, and a third target function; wherein the attention module, the first target function, and the second target function are divided into the defect image branch, and the third target function is divided into the normal image branch; the training processes of the two branches are carried out separately, but on the shared U-Net; and the few-sample defect image generation model is trained by using a hybrid training strategy, as shown in Figure 8 The method comprises:

[0103] In step 601, a plurality of defect tokens and a single product token are combined to obtain an unbalanced abnormal text prompt, and the unbalanced abnormal text prompt and the product-related part in the unbalanced abnormal text prompt are input into the defect image branch and the normal image branch as guide conditions, respectively; The total number of defect categories used is obtained by demand analysis by those skilled in the art.

[0104] In step 602, a fixed number of normal images are selected as inputs for the normal image sample branch, the defect image branch obtains the features of the defect image, and according to the defect image mask corresponding to the defect image, the defect tokens in the unbalanced abnormal text prompt are aligned to the defect area in the image to optimize the first and second target functions; Wherein, the defect image can be any defect image corresponding to a corresponding product of a defect category.

[0105] In step 603, a fixed number of normal images are selected as inputs for the normal image sample branch, the normal image branch obtains the features of the normal image, and the product-related part in the unbalanced abnormal text prompt is used to optimize the third target function; Wherein, the mixed training strategy is used to make defect images of different defect categories participate in training together.

[0106] Wherein, the U-Net encoder and the U-Net decoder can be understood as the U-Net architecture in the diffusion model that needs to be fine-tuned.

[0107] In the defect image branch, the output result of the U-Net encoder is input into the U-Net decoder, and according to the output result of the U-Net decoder and the defect image, the value of the second target function is calculated, and the output result of the U-Net encoder is also input into the attention module, and according to the output result of the attention module and the corresponding defect image mask, the value of the first target function is calculated.

[0108] The output result of the U-Net encoder is input into the U-Net decoder, and according to the output result of the U-Net decoder and the normal image, the value of the third target function is calculated.

[0109] The process of optimizing the first and second target functions and optimizing the third target function can be understood as through multiple iterations, in each iteration, the values of the first, second and third target functions are calculated, and the values of the parameters in the U-Net architecture are adjusted according to the values of the first, second and third target functions, so that the calculated values of the target functions tend to be ideal values, and the specific adjustment process of the parameters is known to those skilled in the art, which will not be described here.

[0110] The embodiment introduces an unbalanced abnormal text prompt in which multiple defect tokens and a single product token are used, and proposes a double-branch diffusion model fine-tuning method under the guidance of the unbalanced abnormal text prompt, that is, using two branches, defect image branch and normal image branch, and further using a mixed training strategy to introduce different categories of defect images, thereby alleviating the overfitting problem caused by the limited number of samples of each defect category, and increasing the diversity of defect images while still maintaining the authenticity of the defect images, so that in the case of fewer defect images of each defect category in the industrial scene, ideal training results can still be achieved, and the trained diffusion model is used to generate defect images.

[0111] It should be noted that the training of the U-Net encoder and the U-Net decoder using the mixed training strategy can be understood as the overall training strategy, and the steps 601-603 can be understood as the actual actions performed in each training.

[0112] In actual application scenarios, the combination of multiple defect tokens and a single product token to obtain an unbalanced abnormal text prompt specifically includes:

[0113] The product token of a single product is combined with the defect tokens of N different defect categories of the product to obtain the unbalanced abnormal text prompt of the product 'a <ob>with <df1> <df2> N >’; wherein <ob>is a product term, <df1> 、 <df2>,..., <df N > are N different defect categories, N is a preset value, N defect tokens of each product correspond to a defect category, N is obtained by demand analysis by those skilled in the art, and in an optional embodiment, N is 4, K*N defect tokens are used to obtain K unbalanced abnormal text prompts of shared product tokens, covering all defect categories of the product, K is a preset value corresponding to K defect categories of a product, and K is obtained by experience analysis by those skilled in the art.

[0114] In an optional embodiment, as shown in Figure 9 , the mixed training strategy is specifically:

[0115] In step 701, defect image sets X are generated according to defect images of all defect categories of the corresponding product. , wherein X is the i-th defect image of the defect category.

[0116] In step 702, a preset number of normal images of the product are selected to generate normal image sets X , wherein X is the i-th normal image.

[0117] In step 703, a first number of defect images are selected from the defect image set X df , a first number of normal images are selected from the normal image set X ob , the selected first number of defect images and the first number of normal images are mixed into a batch of data, and each batch of data is input into the defect image branch and the normal image branch for multi-batch training. Wherein, the first number and the preset number are obtained by experience analysis by those skilled in the art. A batch of data is used for a batch of training, and the specific implementation process of each batch training is as described above in steps 601-603.

[0118] Based on the above defect image branch and normal image branch, the embodiment also provides a specific implementation of each target function. In an optional embodiment, the first target function is , wherein is the cross-attention map corresponding to the n-th defect token, N is the number of defect tokens, L represents the number of layers of the U-Net corresponding to the cross-attention map used in the alignment process, M l is a binary mask with a size of r x r pixels, the defect region value is 1, and the non-defect region value is 0, is the cross-attention map corresponding to the product token, is a pixel-wise multiplication operation. Wherein, is a defect token <df n >the value of the cross-attention at the defective region is increased, is to prevent the product token <ob>Align to the defect area.

[0119] The second objective function is Among them, e df z represents the conditional vector obtained after the unbalanced anomalous text prompt has been processed by the text encoder. df For the defect image x df The latent vector obtained by the VAE encoder, ∈ df For random noise, t df To add noise steps, For use ∈ df and ∈ df For z df The noisy latent vector obtained by adding noise For the predicted noise, The value of the first objective function.

[0120] The third objective function is:

[0121]

[0122] ; where e ob The z-vector represents the conditional vector obtained after the text encoder processes the product-related portion of the unbalanced anomaly text prompt. ob For normal image x ob The latent vector obtained by the VAE encoder, ∈ ob For random noise, t ob To add noise steps, For use ∈ ob and t ob For z ob The noisy latent vector obtained by adding noise, This represents the predicted noise.

[0123] A few-sample defect image generation model is obtained by using the method described in this embodiment. Then, a large number of defect images are generated using the few-sample defect image generation model for training the annotation generation model described in Embodiment 1. The method described in Embodiment 1 is applicable to this embodiment and will not be elaborated here.

[0124] Example 3:

[0125] Based on the method described in Embodiment 1, this invention combines specific application scenarios and uses technical descriptions in relevant scenarios to illustrate the implementation process of the features of this invention in those scenarios.

[0126] like Figure 10 As shown, the industrial defect image pixel-level annotation generation method provided in this embodiment includes the following steps:

[0127] In step 801, double-branch image feature extraction is performed, specifically: a defect image branch and a normal image branch are designed, and the two branches participate in the training of the label generation model at the same time. Each branch extracts discriminative features of the corresponding image through the U-Net in the diffusion model, and uses the features to predict low-resolution pixel-level labels.

[0128] In step 802, for the defect image branch, a feature refinement module is designed to fuse the discriminative features of the U-Net of the defect image and the high-resolution features of the VAE decoder in a cascaded manner to obtain refined features, and the refined features are used to predict high-resolution image pixel-level labels.

[0129] In step 803, image pixel-level label generation is performed, specifically: based on the low-resolution and high-resolution image pixel-level labels of the defect image and the normal image, and the corresponding defect image mask and normal image mask, a label generation objective function is proposed to provide refined pixel-level labels for the generated defect image. The segmentation result of the defect pixels in the defect image is predicted, and all the pixels are divided into two categories: defect pixels and normal pixels.

[0130] There are mainly three implementation parts: 1) double-branch image feature extraction; 2) feature refinement module of the defect image branch; 3) image pixel-level label generation. The following steps are described in detail.

[0131] Among them, the double-branch image feature extraction is specifically: based on the given few-sample defect image generation model, a label generation model based on the U-Net features in the diffusion model and the VAE decoder features is trained, and the defect image branch and the normal image branch are designed in this embodiment. Each branch extracts discriminative features of the corresponding image through the U-Net in the diffusion model, and uses the features to predict low-resolution pixel-level labels, specifically including:

[0132] The output features of the "up-2" and "up-3" layers in the U-Net are compressed in channel number by using a 3x3 convolution block and and

[0133] Upsample F'1 to resolution 64x64 and concatenate F'2.

[0134] Four transformation layers are applied to fuse the concatenated features and obtain unified coarse features

[0135] A 3x3 convolution layer and a softmax function are used to generate low-resolution pixel-level labels based on coarse features ​​

[0136] For the defect image branch, the embodiment designs a feature refinement module, each of which receives two kinds of features, namely the high-resolution features from the VAE and the discriminative features from the U-Net. Then, the discriminative features from the U-Net and the high-resolution features from the VAE decoder are fused in a cascaded manner to gradually refine the coarse features of the defect image. Specifically, it includes:

[0137] The discriminative features from the U-Net are upsampled to align with the high-resolution features from the VAE decoder.

[0138] After 1x1 convolution, the upsampled discriminative features are multiplied element-wise with the high-resolution features from the VAE decoder to enhance the boundary regions of the defects.

[0139] The upsampled discriminative features are processed by two convolutional blocks and added to the features after boundary optimization to maintain the discriminative features.

[0140] The embodiment adopts three cascaded multi-resolution feature refinement modules, i.e., the above operations, to gradually optimize the boundaries of the coarse features Each multi-resolution feature refinement module takes the discriminative features output by the previous multi-resolution feature refinement module as input for refinement, while the first multi-resolution feature refinement module directly uses the coarse features as the initial discriminative input. For the other input of each multi-resolution feature refinement module, the embodiment uses the outputs of the 1st, 2nd, and 3rd "upsampling blocks" in the VAE decoder, respectively.

[0141] The mask prediction module uses a 3x3 convolutional layer and a softmax function to generate refined defect pixel-level labels based on the output of the last multi-resolution feature refinement module The embodiment sets a threshold τ to binarize the second channel of the defect pixel-level labels, thereby obtaining the final defect pixel-level labels.

[0142] After generating the image pixel-level labels, the objective function is calculated, and the objective function is used to optimize the label generation model. The objective function to be optimized by the embodiment is:

[0143]

[0144] where F() represents the calculation of the Focal loss, and represent the low-resolution pixel-level labels of the defect image and the high-resolution pixel-level labels of the defect image, respectively, and represent the low-resolution pixel-level labels of the normal image and the high-resolution pixel-level labels of the normal image, respectively, M df and M′ df M and M' represent the low-resolution ground truth mask of the defect image and the high-resolution ground truth mask of the defect image, respectively ob and M' ob M and M' represent the low-resolution ground truth mask of the normal image and the high-resolution ground truth mask of the normal image, respectively. It is worth noting that in the pixel-level labeling of the normal image, each pixel value is set to 0 to eliminate the potential noise in the normal image.

[0145] The embodiment discloses a pixel-level labeling generation method for industrial defect images. First, the embodiment proposes a double-branch image feature extraction method, designs a defect image branch and a normal image branch, and the two branches participate in the training of the labeling generation model at the same time. Each branch extracts discriminative features of the corresponding image through the U-Net in the diffusion model, and uses the features to predict low-resolution pixel-level labels. Further, for the defect image branch, the embodiment designs a feature refinement module to fuse the U-Net discriminative features of the defect image and the high-resolution features of the VAE decoder in a cascaded manner to obtain refined features, and uses the refined features to predict high-resolution pixel-level labels. Finally, based on the low-resolution and high-resolution pixel-level labels of the defect image and the normal image, a labeling generation objective function is proposed. The embodiment uses defect images and normal images for training to provide refined pixel-level labels for generated defect images.

[0146] The few-sample defect image generation model needs to be trained in advance using a small amount of defect images and normal images, as shown in Figure 11 The training process of the few-sample defect image generation model includes the following steps:

[0147] In step 901, a small amount of defect images and corresponding defect image masks are selected as inputs for the defect image branch, and a fixed number of normal images are selected as inputs for the normal image sample branch.

[0148] In step 902, an unbalanced abnormal text prompt is designed, specifically: a plurality of defect tokens and a single product token suitable for an industrial scene are combined to obtain an unbalanced abnormal text prompt as a guide condition to control the diffusion model image generation.

[0149] In step 903, the defect region is aligned, specifically: the defect image branch is designed to obtain the features of the defect image. Based on the defect image mask corresponding to the defect image, the defect tokens in the unbalanced abnormal text prompt are aligned to the defect region in the image through an attention module; and a noise prediction objective function under the unbalanced abnormal text prompt is optimized.

[0150] In step 904, normal image fine-tuning, specifically: design normal image branch, obtain the features of normal images. Use the product-related part in the unbalanced abnormal text prompt to optimize the noise prediction objective function.

[0151] In step 905, mixed training strategy, specifically: fine-tune a unified U-Net for each product containing different defect categories. The defect image set includes a small number of defect images of all defect categories of the same product, and the normal image set includes a fixed number of normal images of the product. Each time, an equal number of images are extracted from the defect image set and the normal image set, and mixed for training as a batch.

[0152] The unbalanced abnormal text prompt design specifically includes: adopting an unbalanced abnormal text prompt 'a <ob>with <df1> <df2> N >’, wherein <ob>is a product term, <df1> <df2>…<df N > are N defect tokens. The unbalanced abnormal text prompt of each product is composed of the single product token of the product and N defect tokens.

[0153] The defect region alignment specifically comprises: aligning the defect region as the foreground from the defect picture separately with the defect token <df n > through a cross-attention mechanism, and optimizing a noise prediction objective function under the unbalanced abnormal text prompt. As shown in Figure 12 , specifically comprising:

[0154] In step 1001, based on the features output by the U-Net encoder and the defect image mask, the defect token <df n > is aligned to the defect region in the defect image using a cross-attention module to optimize the first objective function:

[0155]

[0156] Wherein, is the cross-attention map corresponding to the nth defect token <df n >, and N is the number of defect tokens. L represents the number of layers of the U-Net corresponding to the cross-attention map used in the alignment process, and M l is a binary mask with a size of r x r pixels, and the value of the defect region is 1 and the value of the non-defect region is 0. is the product token <ob>The corresponding cross-attention map, is a pixel-wise multiplication. This objective function decouples the defect region from the defect image, the first term will force the model to focus on the defect region, and the second term will prevent the model from focusing on the product region n The value of cross-attention in the defect region is increased, and the second term is to prevent the model from focusing on the product region <ob>Align to defect area.

[0157] In step 1002, the defect image branch uses the unbalanced anomaly text cue 'a <ob>with <df1> <df2>…<df N >’,optimizing the second objective function of the fine-tuned U-Net:

[0158]

[0159] where e df represents the conditional vector obtained after the unbalanced abnormal text prompt is input into the CLIP text encoder, the defect image x df is input into the VAE encoder to obtain the latent vector z df , and the random noise ∈ df and the number of noise addition steps t df are used to add noise to z df to obtain the noisy latent vector Then, under the guidance of the condition e df , the noise is predicted, and the predicted noise and ∈ df are calculated to obtain the loss, which is used to fine-tune the U-Net.

[0160] The normal image fine-tuning specifically includes: the normal image branch uses the part 'a <ob>the third objective function of the U-Net is optimized:

[0161]

[0162] where c ob denotes the product-related part 'a <ob>conditional vector obtained after passing the CLIP text encoder on the normal image x ob latent vector z obtained by the VAE encoder ob and using random noise ∈ ob and the number of noise adding steps t ob z ob is added with noise to obtain the noisy latent vector Then, noise prediction is performed under the guidance of the condition e ob and the predicted noise and ∈ ob is calculated to fine-tune the U-Net.

[0163] The hybrid training strategy specifically comprises: using the hybrid training strategy to fine-tune a unified U-Net for each product. Given a product image set, which covers 15 products, each product corresponds to a defect image with multiple defect categories, a corresponding defect image mask and a normal image. All defect images of all products are summarized into a unified defect image set For each defect category, an unbalanced abnormal text prompt composed of different defect tokens is used. In addition, a fixed number of normal images are extracted to form a normal image set In each fine-tuning step, the same number of images are extracted from X df and X ob and combined into a batch for training. Not only does it alleviate the overfitting problem that may be caused by the limited number of samples for each defect category, but it also increases the diversity of defect images while still maintaining the authenticity of defect images.

[0164] The U-Net architecture obtained through the above training process is used in a diffusion model for image generation under specific conditions. Given an input image x0, the diffusion model first converts it into a representation in the latent space z = ε(x0), then adds randomly sampled noise ∈ ~ N(0, I) to it to obtain where t is a randomly selected time step. Next, the U-Net is used to predict the noise ∈. Let c θ (P) be the CLIP text encoder that maps the conditional text prompt P to the conditional vector e. The noise prediction objective function that the diffusion model needs to optimize is:

[0165]

[0166] where is the predicted noise.

[0167] The embodiment proposes an unbalanced abnormal text prompt for industrial defect image generation, which contains multiple defect tokens and a single product token. Further, under the guidance of the unbalanced abnormal text prompt, a double-branch diffusion model fine-tuning method is proposed. For the defect image branch, the invention proposes a defect region alignment objective function to align the defect token and the defect region, and optimizes the noise prediction objective function under the unbalanced abnormal text prompt to realize high diversity of defects. For the normal image branch, only the product-related part in the unbalanced abnormal text prompt is used to optimize the noise prediction objective function to synthesize product images with global consistency and local changes. The training processes of the two branches are carried out separately, but are implemented on a shared U-Net, as shown in Figure 13 representing each unbalanced abnormal text prompt, the defect image branch and the normal image branch act on the U-Net architecture. Moreover, the embodiment adopts a hybrid training strategy to fine-tune a unified U-Net for each product. Thus, a large number of defect images with high authenticity and diversity are generated by training with a small number of defect images and some normal images. The generated defect images are used for training the above-mentioned annotation generation model, thereby obtaining the trained annotation generation model for image high-resolution pixel-level annotation.

[0168] Embodiment 4:

[0169] As Figure 14 shown, is the architecture schematic diagram of the industrial defect image pixel-level annotation generation device of the embodiment of the invention. The industrial defect image pixel-level annotation generation device of the embodiment includes one or more processors 21 and a memory 22. Among them, Figure 14 The processor 21 is taken as an example.

[0170] The processor 21 and the memory 22 can be connected through a bus or other means, Figure 14 The connection through the bus is taken as an example.

[0171] The memory 22 is a kind of non-volatile computer readable storage medium, which can be used to store non-volatile software programs and non-volatile computer executable programs, such as the industrial defect image pixel-level annotation generation method in embodiment 1. The processor 21 executes the non-volatile software programs and instructions stored in the memory 22, thereby executing the industrial defect image pixel-level annotation generation method.

[0172] ​The memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 22 can optionally include a memory disposed remotely from the processor 21, which can be connected to the processor 21 through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0173] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, perform the industrial defect image pixel-level labeling generation method in the above-mentioned embodiment 1.

[0174] It is worth noting that the information interaction, execution process, and the like between the modules and units in the above-mentioned apparatus and system are based on the same concept as the processing method embodiments of the present application, and the specific content can be referred to the description in the method embodiments of the present application, which will not be described here.

[0175] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer-readable storage medium, which can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like.

[0176] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.< / ob> < / ob> < / df1> < / ob> < / ob> < / ob> < / df1> < / ob> ​ < / df1> < / ob> < / ob> < / df1> < / ob> ​ < / df1> < / ob>

Claims

1. A method for generating pixel-level annotations for industrial defect images, characterized in that, The annotation generation model is trained, and the trained model is used to generate pixel-level annotations. The annotation generation model includes a VAE encoder, a U-Net encoder, a U-Net decoder, a VAE decoder, a feature refinement module, and a mask prediction module. It is further divided into a defective image branch and a normal image branch. The VAE encoder, U-Net encoder, U-Net decoder, VAE decoder, feature refinement module, and mask prediction module constitute the defective image branch, while the VAE encoder, U-Net encoder, U-Net decoder, VAE decoder, and mask prediction module constitute the normal image branch. The training of the annotation generation model specifically includes: The defect image branch extracts discriminative features from the defect image and uses these discriminative features to perform low-resolution pixel-level annotation on the defect image. In the defect image branch, the defect image is input to the VAE encoder, the output of the VAE encoder is input to the U-Net encoder, the output of the U-Net encoder is input to the U-Net decoder, the output of the U-Net decoder is input to the VAE decoder, and the output of the U-Net decoder is the discriminative features of the defect image. The normal image branch extracts discriminative features from the normal image and uses these discriminative features to perform low-resolution pixel-level annotation on the normal image. In the normal image branch, the normal image is input to the VAE encoder, the output of the VAE encoder is input to the U-Net encoder, the output of the U-Net encoder is input to the U-Net decoder, and the U-Net decoder outputs the discriminative features of the normal image. The defect image branch fuses the discriminative features of the defect image with the high-resolution features output by the VAE decoder to obtain refined features, and uses the refined features to perform high-resolution pixel-level annotation on the defect image. Upsampling is performed on the low-resolution pixel-level annotations of the normal image to obtain high-resolution pixel-level annotations of the normal image; Based on the low-resolution pixel-level annotations of the defective image, the low-resolution pixel-level annotations of the normal image, the high-resolution pixel-level annotations of the defective image, the high-resolution pixel-level annotations of the normal image, the mask of the defective image, and the mask of the normal image, an objective function is generated, and the annotation generation model is optimized using the objective function. The discriminative features of the defect image include a first feature. Second feature The first feature of the U-Net decoder output is processed using 3×3 convolutional blocks. Channel number compression is performed to obtain the first compression feature. The second feature of the U-Net decoder output is processed using convolutional blocks. Channel number compression is performed to obtain the second compression feature. ; wherein, the first feature The second feature is the output of the second layer of the U-Net decoder. Features of the third-level output of the U-Net decoder; for the first compressed features Perform upsampling to compress the first feature. The upsampling result and the second compression feature By splicing the pieces together, we can obtain the splicing features. ; regarding the splicing features Applying multiple transformation layers yields coarse features. ; in the roughness feature A 3×3 convolutional layer and a softmax function are applied to generate low-resolution pixel-level annotations for defect images; The refined features are processed using a 3×3 convolutional layer and a softmax function to generate pixel-level defect annotations. Use preset threshold The defect pixel-level annotation Binarization is performed to obtain high-resolution pixel-level annotations of the defect image; The objective function is: ; in, This represents the calculation of Focal loss. and These represent the low-resolution pixel-level annotations and high-resolution pixel-level annotations of the defect image, respectively. and These represent low-resolution pixel-level annotations and high-resolution pixel-level annotations of a normal image, respectively. and These represent the low-resolution ground truth mask and the high-resolution ground truth mask of the defect image, respectively. and These represent the low-resolution true mask and the high-resolution true mask of a normal image, respectively.

2. The method for generating pixel-level annotations of industrial defect images according to claim 1, characterized in that, The high-resolution features include the first High-resolution features It is an integer. Greater than 0 and Less than 4; the process of fusing the discriminative features of the defect image and the high-resolution features output by the VAE decoder to obtain refined features specifically includes: In sequence, the first Level features and the first High-resolution features are fused to obtain the first Level 4 features are used as the refined features; wherein, the discriminative features of the defect image are used as Level 1 features; The first Level features and the first High-resolution features are fused to obtain the first Level features, specifically including: The first Level features and the first High-resolution feature alignment; For the After performing 1×1 convolution and upsampling on the first-level features sequentially, they are then aligned with the corresponding first-level features. High-resolution features are multiplied element-wise to obtain the first... Optimize boundary features; The first After the first-level features are processed by two convolutional blocks, they are combined with the second-level features. The optimized boundary features are added together to obtain the first... Level features.

3. The method for generating pixel-level annotations of industrial defect images according to claim 1, characterized in that, The defective image is pre-generated using a few-shot defective image generation model, which includes the U-Net encoder as described in claim 1, the U-Net decoder as described in claim 1, an attention module, a first objective function, a second objective function, and a third objective function; wherein the attention module, the first objective function, and the second objective function are assigned to the defective image branch, and the third objective function is assigned to the normal image branch; the training processes of the two branches are performed separately, but are implemented on a shared U-Net; a hybrid training strategy is used to train the few-shot defective image generation model, specifically including: Multiple defective words and a single product word are combined to obtain an unbalanced anomaly text prompt. The unbalanced anomaly text prompt and the product-related part of the unbalanced anomaly text prompt are used as guiding conditions and input into the defective image branch and the normal image branch respectively. A few sample defect images and their corresponding defect image masks are selected as inputs to the defect image branch. The defect image branch acquires the features of the defect images and aligns the defect words in the unbalanced abnormal text prompts to the defect regions in the images according to the defect image mask corresponding to the defect images, so as to optimize the first objective function and the second objective function. A fixed number of normal images are selected as input to the normal image sample branch, which acquires the features of the normal images and uses the product-related parts of the imbalanced abnormal text prompts to optimize the third objective function; wherein, the hybrid training strategy is used to enable defect images of different defect categories to participate in training together.

4. The method for generating pixel-level annotations of industrial defect images according to claim 3, characterized in that, The first objective function is ;in, This is the first Cross-attention map corresponding to each defective word. It is the number of defective lexical units. This indicates the number of U-Net layers corresponding to the cross-attention maps used in the alignment process. It is a binary mask with a size of For pixels, the value is 1 for defective areas and 0 for non-defective areas. It is a cross-attention graph corresponding to product keywords. To perform multiplication operations on a pixel basis; The second objective function is ;in, This represents the conditional vector obtained after the imbalanced anomalous text prompt has been processed by the text encoder. For defect images Latent vectors obtained through the VAE encoder It is random noise. To add noise steps, For use and right The noisy latent vector obtained by adding noise , This represents the predicted noise.

5. The method for generating pixel-level annotations of industrial defect images according to claim 3, characterized in that, The third objective function is: ;in, This represents the conditional vector obtained after the text encoder processes the product-related portion of the imbalanced anomaly text prompt. Normal image Latent vectors obtained through the VAE encoder It is random noise. To add noise steps, For use and right The noisy latent vector obtained by adding noise, This represents the predicted noise.

6. A pixel-level annotation generation device for industrial defect images, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the industrial defect image pixel-level annotation generation method according to any one of claims 1-5.

7. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions that are executed by one or more processors to perform the method described in any one of claims 1-5.

Images