Graph sample generation method and device, equipment, readable storage medium and program product

Abstract

The invention discloses a graph sample generation method and device, equipment, a readable storage medium and a program product, and belongs to the field of artificial intelligence. According to the embodiment of the invention, a graph sample, description information of the graph sample and a design graph are obtained, and a historical graph sample including defect types, defect areas and defect shapes is obtained based on the design graph; determining a target defect area with a target defect type and a target defect shape in the design drawing based on the historical graph sample; according to the target defect area and the target defect shape, adding a defect mask in the design drawing to obtain a defect mask design drawing; inputting the graph sample, the description information and the defect mask design graph into a graph sample generation model, adding noise to the graph sample through the graph sample generation model to obtain a noise graph, and performing noise reduction processing on the noise graph based on the noise graph, the description information and the defect mask design graph to obtain a target graph sample, the richness and authenticity of training samples of the defect recognition model can be improved, and the recognition accuracy of the defect recognition model is further improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence, and in particular, relates to a method, apparatus, device, computer-readable storage medium, and computer program product for generating a graph sample. Background Art

[0002] Machine vision defect recognition refers to the use of computer vision technology to detect and identify defects on the surface of products. Machine vision defect recognition systems are often used to monitor production lines in real time during product production and automatically detect unqualified products, thereby reducing the cost of manual inspection and improving detection accuracy. In order to better improve the recognition efficiency and accuracy of machine vision defect recognition systems, it is necessary to train them with a large number of defect image samples. However, it is difficult to obtain a large number of defect image samples in actual production, so insufficient defect image samples have always been one of the bottlenecks restricting the application of machine vision defect recognition.

[0003] In the machine vision defect recognition method of the prior art, defect image samples are generally generated by manual drawing and AI synthesis to train the machine vision defect recognition model. Among them, the method of manually drawing to generate defect image samples is inefficient and the generated defect image samples are not realistic enough, while the method of AI synthesis to generate defect image samples also requires manual drawing of defect masks, which cannot be fully automated and the generated defect image samples are not rich enough, which leads to low recognition accuracy of the trained machine vision defect recognition model. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device, computer-readable storage medium, and computer program product for generating a graph sample, which can increase the richness and authenticity of trap recognition model training samples, thereby improving the recognition accuracy of the trap recognition model.

[0005] In a first aspect, an embodiment of the present application provides a method for generating a graph sample, the method comprising:

[0006] Obtaining image samples, description information of the image samples, design drawings of the image samples, and historical image samples obtained based on the design drawings, wherein the historical image samples include defect categories, defect area information, and defect shapes;

[0007] Determine target defect area information and target defect shape of target defect type in the design drawing based on historical drawing samples;

[0008] According to the target defect area information and the target defect shape, a defect mask is added to the design diagram to obtain a defect mask design diagram;

[0009] The image sample, description information and defect mask design drawing are input into the image sample generation model, noise is added to the image sample through the image sample generation model to obtain a noise image, and the noise image is denoised based on the noise image, description information and defect mask design drawing to obtain a target image sample.

[0010] In an achievable implementation, before obtaining the description information of the image sample, the method further includes:

[0011] Convert image samples into image embedding vectors;

[0012] Calculate the similarity between the image embedding vector and the text embedding vector of the historical description information;

[0013] The historical description information corresponding to the highest similarity is selected as the description information of the graph sample.

[0014] In an achievable implementation, determining a target defect area having a target defect type in a design drawing based on a historical drawing sample includes:

[0015] Determine all defect areas of the design drawing that have a target defect type from the historical drawing sample;

[0016] A defect area that meets a preset condition is determined from all defect areas as a target defect area.

[0017] In an achievable implementation, determining a target defect shape of a target defect type in a design drawing based on a historical drawing sample includes:

[0018] Determine all defect shapes of the target defect type in the design drawing from the historical drawing sample;

[0019] An original defect shape is randomly selected from all defect shapes, and the shape edge of the original defect shape is sampled to obtain sampling points;

[0020] Curve fitting is performed based on the sampling points to obtain the target defect shape.

[0021] In an achievable implementation, the image sample generation model includes a U-Net network, and the noise image is subjected to denoising based on the noise image, the description information, and the defect mask design image to obtain a target image sample, including:

[0022] Input the noise map, description information and defect mask design map into the U-Net network;

[0023] Convert the noise map into the hidden state of the U-Net network;

[0024] The cross-attention mechanism is adopted through the U-Net network to calculate the predicted noise based on the description information and the defect mask design diagram;

[0025] The predicted noise is used to denoise the noise image and obtain the target image sample.

[0026] In an achievable implementation, the noise map, description information, and defect mask design map are input into a U-Net network, the noise map is converted into a hidden state of the U-Net network, and a cross-attention mechanism is adopted through the U-Net network. The formula for calculating the predicted noise based on the description information and the defect mask design map includes:

[0027]

[0028] Among them, ∈θ(zt, t, τ θ(y)) represents the predicted noise, zt represents the hidden representation of the noise image at time step t, and τ θ(y) represents the intermediate representation of the encoded conditional input y. represents the hidden state of the U-Net network at layer i, Q represents the query vector, represents the query matrix, K represents the key vector, represents the weight matrix, K represents the value vector, represents the weight matrix, D UNet Represents a neural network based on the U-Net architecture;

[0029] The formula for denoising the noise image using the predicted noise to obtain the target image sample includes:

[0030]

[0031] Among them, zt-1 represents the target image sample after denoising, zt represents the noise image at time step t, αt and βt represent the coefficients related to time step t, αt represents the proportion of the image retained during the forward diffusion process, and βt represents the noise image at time step t. t represents the proportion of noise added during the forward diffusion process, represents the cumulative product of α values ​​at all previous time steps, and ∈θ represents the prediction noise.

[0032] In a second aspect, an embodiment of the present application provides a device for generating a graph sample, the device comprising:

[0033] The acquisition module is used to acquire image samples, description information of image samples, design drawings of image samples, and historical image samples obtained based on the design drawings. The historical image samples include defect categories, defect area information, and defect shapes.

[0034] The determination module is used to determine the target defect area information and target defect shape of the target defect type in the design drawing based on the historical drawing sample.

[0035] The adding module is used to add a defect mask to the design drawing according to the target defect area information and the target defect shape to obtain a defect mask design drawing.

[0036] The generation module is used to input the image sample, description information and defect mask design diagram into the image sample generation model, add noise to the image sample through the image sample generation model to obtain a noise image, and perform denoising on the noise image based on the noise image, description information and defect mask design diagram to obtain a target image sample.

[0037] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement a method for generating a sample image as in any one of the embodiments of the first aspect.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, a method for generating a graph sample as in any one of the embodiments of the first aspect is implemented.

[0039] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes a method for generating a sample image as described in any one of the embodiments of the first aspect.

[0040] The image sample generation method, device, equipment, computer-readable storage medium and computer program product of the embodiment of the present application first obtain an image sample, description information of the image sample and a design drawing of the image sample, and a historical image sample obtained based on the design drawing, wherein the historical image sample includes a defect category, defect area information and a defect shape; then, based on the historical image sample, target defect area information and a target defect shape of a target defect type in the design drawing are determined, and according to the target defect area information and the target defect shape, a defect mask is added to the design drawing to obtain a defect mask design drawing; a brand new defect mask can be obtained by randomly selecting a target defect type and determining a defect area and a defect shape according to the defect type. The design drawing provides a variety of choices for the generation of defect image samples; finally, the image sample, description information and defect mask design drawing are input into the image sample generation model, and noise is added to the image sample through the image sample generation model to obtain a noise image. The noise image is denoised based on the noise image, description information and defect mask design drawing to obtain the target image sample; after adding noise to the image sample through the image sample generation model, the description information and defect mask design drawing are used to predict the noise and eliminate the noise added by the image sample, so as to obtain a new sample image that is similar to but different from the sample image, which increases the richness and authenticity of the training samples of the trap recognition model, thereby improving the recognition accuracy of the trap recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0042] Figure 1 It is a flowchart of a method for generating a graph sample provided by an embodiment of the present application;

[0043] Figure 2 is a schematic diagram of a defect image generated by a defect image generation model provided by an embodiment of the present application;

[0044] Figure 3 It is a schematic diagram of the architecture of a stable diffusion model provided by an embodiment of the present application;

[0045] Figure 4 is a structural schematic diagram of a graph sample generating device provided by an embodiment of the present application;

[0046] Figure 5 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0048] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0049] Machine vision defect recognition refers to the use of computer vision technology to detect and identify defects on the surface of products. Machine vision defect recognition systems are often used to monitor production lines in real time during product production and automatically detect unqualified products, thereby reducing the cost of manual inspection and improving detection accuracy. In order to better improve the recognition efficiency and accuracy of machine vision defect recognition systems, it is necessary to train them with a large number of defect image samples. However, it is difficult to obtain a large number of defect image samples in actual production, so insufficient defect image samples have always been one of the bottlenecks restricting the application of machine vision defect recognition.

[0050] In the machine vision defect recognition method of the prior art, defect image samples are generally generated by manual drawing and AI synthesis to train the machine vision defect recognition model. Among them, the method of manually drawing to generate defect image samples is inefficient and the generated defect image samples are not realistic enough, while the method of AI synthesis to generate defect image samples also requires manual drawing of defect masks, which cannot be fully automated and the generated defect image samples are not rich enough, which leads to low recognition accuracy of the trained machine vision defect recognition model.

[0051] In the embodiment of the present application, a defect mask (Defect Mask) in the field of machine vision and image processing refers to a binary image used to mark or annotate defective areas in an image. Typically, the pixel value of the defective area in the defect mask is 1 (white), and the pixel value of the non-defective area is 0 (black). A design drawing refers to a design drawing of a product generated by computer software to assist in the planning, management and control of the manufacturing process (Computer-Aided Manufacturing, CAM), which contains the structural layout, component location and other important design information of the product, and can also be referred to as a CAM drawing.

[0052] In order to solve the problems of the prior art, the embodiments of the present application provide a method, device, equipment, computer storage medium and computer program product for generating a graph sample. The graph sample generating method provided by the embodiments of the present application is first introduced below.

[0053] Figure 1 FIG. 1 is a flow chart of a method for generating a sample image provided by an embodiment of the present application. Figure 1 As shown, the method includes steps: S110 to S140.

[0054] The image sample generation method of this embodiment can be applied to the scenario of printed circuit board (PCB) defect recognition, and the defect recognition model is trained by generating defect samples.

[0055] S110: Acquire a graph sample, description information of the graph sample, a design drawing of the graph sample, and a historical graph sample obtained based on the design drawing, wherein the historical graph sample includes defect category, defect region information, and defect shape.

[0056] The image samples may include defect-free images, which may also be called OK images, i.e. images of defect-free printed circuit boards. Defective images refer to defective images, which may be called NG (Not Good) images, i.e. defect image samples used to train defect recognition models. The description information refers to a detailed description or explanation of the defect-free image, and the description information may be information in any form of text, i.e. text description information. For example, for an image containing a dog and a cat, the text description information may be a dog and a cat playing on the grass. Each CAM image has an OK image and an NG image corresponding to it, and the corresponding CAM image and NG image can be matched through the OK image. The historical image samples obtained based on the design image refer to a sample set that is classified from the historical NG image corresponding to the CAM image, including defect categories, defect area information, defect shapes, and the corresponding relationship between them. The defect area information refers to the area where a defect type may appear in the corresponding CAM image, and the defect area is obtained by classifying and summarizing the existing defect images. The defect shape refers to the possible graphic shape of a defect type in the corresponding CAM image, and the defect shape is obtained by annotating the existing NG image using a polygon tool. Exemplarily, a defect category is randomly selected, and the area and shape that may appear in the CAM image can be determined in the historical image sample corresponding to the CAM image.

[0057] An OK image sample, text description information of the OK image sample, a CAM image corresponding to the OK image sample, and a history image sample obtained based on the CAM image are obtained. The history image sample includes defect category, defect area information and defect shape and the corresponding relationship between them.

[0058] S120: Determine target defect region information and target defect shape of the target defect type in the design drawing based on the historical drawing sample.

[0059] A defect type is randomly selected from the history image sample of the CAM image corresponding to the OK image sample, and a defect area and a defect shape are selected as the position and shape of the defect occurrence from the area and shape in which the defect type may appear in the CAM image based on the history image sample.

[0060] S130: Adding a defect mask to the design drawing according to the target defect region information and the target defect shape to obtain a defect mask design drawing.

[0061] According to the selected defect position and defect shape, the defect shape is pasted into the acquired CAM image to form a CAM image containing the defect, and the CAM image containing the defect is converted into a defect mask form to finally obtain a defect mask CAM image containing the defect mask.

[0062] S140: Input the image sample, description information and defect mask design image into the image sample generation model, add noise to the image sample through the image sample generation model to obtain a noise image, perform denoising on the noise image based on the noise image, description information and defect mask design image, and obtain a target image sample.

[0063] The image sample generation model can generate NG images based on OK image samples, text description information and defect mask CAM images. As training image samples for training defect recognition models, it is mainly used to generate NG images in this embodiment and can also be called a defect image generation model.

[0064] The OK image sample and its corresponding text description information and defect mask CAM image are input into the defect image generation model. In the defect image generation model, noise is randomly added to the OK image sample to obtain a noise image. Exemplarily, the randomly added noise can be Gaussian noise. In the defect image generation model, the noise image is denoised based on the noise image, text description information and defect mask CAM image to obtain an NG image sample.

[0065] Figure 2 FIG. 1 is a schematic diagram showing a defect image generated by a defect image generation model provided by an embodiment of the present application. Figure 2 As shown, marker 1 represents an OK image sample, marker 2 represents a defect mask CAM image, and marker 3 represents an NG image sample. The OK image sample and its corresponding text description information and defect mask CAM image are input into the defect image generation model, and an NG image sample can be output.

[0066] In one embodiment, the image sample generation model can generate a new OK image sample based on the OK image sample and its corresponding text description information. The newly generated OK image sample is similar to the original OK image sample but has certain differences. In this way, a rich variety of OK image samples can be generated. The OK image samples are used as defect-free image samples for training the defect recognition model.

[0067] The image sample generation method, device, equipment, computer-readable storage medium and computer program product of the embodiment of the present application first obtain an image sample, description information of the image sample and a design drawing of the image sample, and a historical image sample obtained based on the design drawing, wherein the historical image sample includes a defect category, defect area information and a defect shape; then, based on the historical image sample, target defect area information and a target defect shape of a target defect type in the design drawing are determined, and according to the target defect area information and the target defect shape, a defect mask is added to the design drawing to obtain a defect mask design drawing; a brand new defect mask can be obtained by randomly selecting a target defect type and determining a defect area and a defect shape according to the defect type. The design drawing provides a variety of choices for the generation of defect image samples; finally, the image sample, description information and defect mask design drawing are input into the image sample generation model, and noise is added to the image sample through the image sample generation model to obtain a noise image. The noise image is denoised based on the noise image, description information and defect mask design drawing to obtain the target image sample; after adding noise to the image sample through the image sample generation model, the description information and defect mask design drawing are used to predict the noise and eliminate the noise added by the image sample, so as to obtain a new sample image that is similar to but different from the sample image, which increases the richness and authenticity of the training samples of the trap recognition model, thereby improving the recognition accuracy of the trap recognition model.

[0068] In one embodiment, before obtaining the description information of the image sample in step S110, the method further includes: converting the image sample into an image embedding vector, calculating the similarity between the image embedding vector and the text embedding vector of the historical description information, and selecting the historical description information corresponding to the highest similarity as the description information of the image sample.

[0069] The description information of the image sample may include text description information, and the description information of the image sample can be obtained through the contrastive language-image pre-training model (CLIP). The text description information of the historical data is input into the CLIP model text encoder in advance to obtain the text embedding vector of each text description, and the text description information of the historical data is used as the candidate text description to describe the OK image sample. The OK image sample is input into the CLIP model image encoder, and the OK image sample is converted into an image embedding vector through the CLIP model. The cosine similarity can be used to calculate the similarity between the image embedding vector and each text description embedding vector. The historical description information with the highest similarity is selected as the text description information of the OK image sample.

[0070] This embodiment converts the image sample into an image embedding vector, calculates the similarity between the image embedding vector and the text embedding vector of the historical description information, and selects the historical description information corresponding to the highest similarity as the description information of the image sample. This ensures that the description information is highly relevant to the image sample, avoids the subjectivity and errors that may be caused by manual annotation, and improves the accuracy and reliability of the description information.

[0071] In one embodiment, determining the target defect area of ​​the target defect type in the design drawing based on the historical image samples in step S120 includes: determining all defect areas of the target defect type in the design drawing from the historical image samples; and determining a defect area that meets preset conditions from all defect areas as the target defect area.

[0072] After randomly selecting a defect type from the history map sample of the CAM map corresponding to the OK map sample, a potential area set is obtained in the area where the defect type may appear in the CAM map based on the history map sample, and the CAM map and the area where defects may appear in the CAM map are converted into a binary mask, where the pixel value of the area where defects may appear in the CAM map is 1, represented by white, and the pixel value of the non-defective area in the CAM map is 0, represented by black. A connected component refers to a group of pixels with the same attributes in the image, which are connected to each other in a specific neighborhood. Determine the defect area that meets the preset conditions from the area where defects may appear in the CAM map as the target defect area. The potential defect mask refers to the defect mask corresponding to the area where defects may appear in the CAM map. The connected component of the defect area refers to the pixels in the binary mask with a pixel value of 1 and adjacent to each other as a connected component.

[0073] Based on the requirements of some defects on the surrounding image information, the preset condition can be to set a minimum and maximum area threshold to exclude areas where the area of ​​the potential defect mask is too small or too large in the area where defects may occur in the CAM image. The preset condition can also be to set a threshold for the number of connected components to exclude areas where the number of connected components is too large or too small in the area where defects may occur in the CAM image. Some defects have requirements for the connected components themselves. The set condition can also be to further filter the area of ​​each defective connected component to exclude areas where the defective connected components whose area is outside the preset threshold range. The set condition can also be to filter the location of the defective connected components according to the requirements of the defect type. For example, the defect of a poor solder joint should be located in the solder joint area, and the surface scratches should be located in the surface area. Through at least one of the above preset conditions, the areas that do not meet the requirements are screened out in the area where defects may occur in the CAM image, and then an area is randomly selected from the screened areas as the location where the defect occurs.

[0074] This embodiment sets specific preset conditions to screen the target defect areas, which can ensure that only those areas that truly meet the specific defect characteristics will be selected. This helps to eliminate interference information that does not meet the conditions, improve the authenticity of the defect image samples, and improve the accuracy of defect identification.

[0075] In one embodiment, determining the target defect shape of the target defect type in the design drawing based on the historical image samples in step S120 includes: determining all defect shapes of the target defect type in the design drawing from the historical image samples; randomly selecting an original defect shape from all defect shapes, and sampling the shape edges of the original defect shape to obtain sampling points; and performing curve fitting based on the sampling points to obtain the target defect shape.

[0076] After randomly selecting a defect type from the history image sample of the CAM image corresponding to the OK image sample, randomly select an original defect shape from the possible shapes of the defect type in the CAM image determined based on the history image sample, then sample the shape edge of the original defect shape to obtain sampling points, and perform curve fitting based on the sampling points to obtain the target defect shape as the final defect shape. The above process can be completed by Bezier curves, which are a parametric curve widely used in computer graphics to generate smooth curves. It is defined by a set of control points, and the shape of the curve can be adjusted by changing the positions of these control points.

[0077] The boundary points of the selected original defect shape are uniformly sampled to generate a series of control points, and the Bezier curve is used to fit the sampled control points to generate a new shape as the final defect shape. Among them, the degree of the Bezier curve can be selected, and the commonly used ones are linear (1st), quadratic and cubic Bezier curves.

[0078] This embodiment samples the shape edges of the actual defect shapes in the historical image samples and performs curve fitting based on the sampling points to more accurately capture the detailed features of the defect shape. The generated target defect shape is closer to the morphological features of the actual defect, thereby improving the richness and authenticity of the defect image.

[0079] In one embodiment, the image sample generation model in step S140 includes a U-Net network, and denoising is performed on the noise image based on the noise image, description information and defect mask design image to obtain a target image sample, including: inputting the noise image, description information and defect mask design image into the U-Net network; converting the noise image into a hidden state of the U-Net network; using a cross-attention mechanism through the U-Net network to calculate predicted noise based on the description information and the defect mask design image; and using the predicted noise to denoise the noise image to obtain a target image sample.

[0080] The graph sample generation model is built based on the stable diffusion model. Figure 3 FIG. 1 shows a schematic diagram of the architecture of a stable diffusion model provided by an embodiment of the present application. Figure 3 As shown, the stable diffusion architecture includes pixel space, latent space and control conditions. The latent space includes the U-Net network, ε is the encoder, x is the original image, z T is the latent representation of the noisy image, τθ is the domain-specific encoder, For the decoder, For the generated image, QKV represents the cross attention mechanism, the dotted line represents the jump connection, and y represents the control condition. The control condition can be text, semantic graph, etc. In the embodiment of the present application, the control condition is text description information and defect mask CAM graph.

[0081] The encoder receives the original image as input, and through a series of convolution operations, gradually reduces the spatial resolution of the image (downsampling), while increasing the number of channels of the feature map, and finally outputs a latent space representation. This process can be regarded as a process of mapping from a high-dimensional image space to a low-dimensional latent space. The decoder receives the latent space representation generated by the encoder as input, and through a series of inverse convolution operations (upsampling), gradually restores the spatial resolution of the image, and finally generates a new image of the same size as the input image. This process is the process of mapping from a low-dimensional latent space back to a high-dimensional image space. The core idea of ​​the diffusion process is to gradually add noise to the input data during the training process until the data becomes completely random, and then train a model to reverse this process, that is, gradually recover the original data from pure noise. This reversal process is called a denoising process. In Stable Diffusion, the diffusion process is implemented by a UNet network with time as the input condition. In this embodiment, the UNet network is trained to predict the noise at a specific time step so as to recover the original image from the noise. Conditional control is to make the generated image more in line with specific needs, such as generating images with specific defects. The Stable Diffusion model can guide the image generation process by introducing conditional control. Exemplarily, text descriptions, defect masks, etc. are input into the model as conditions. To achieve this, the Stable Diffusion model utilizes a cross-attention mechanism to enhance its backbone network (UNet), which allows the model to take into account additional conditional information when generating images, so that the generated images are more consistent with expectations. The training goal of the denoising diffusion model is to optimize the model parameters by minimizing the difference between the predicted noise and the actual noise added. Specifically, the model attempts to predict the noise added at each time step, and the training goal is to minimize the mean squared error between these predicted noises and the actual noise.

[0082] Input an original image in RGB space The encoder ε encodes x into a latent space representation z = ε(x), reconstructs the image from the latent space, and generates the image in R indicates that the pixel values ​​of the image are in the real number range, H and WW indicate the height and width of the original input image, and c indicates the number of channels in the latent space representing z. The encoder downsamples the image with a factor of f = H / h = W / w. With the perceptual compression model ε and It has an efficient and low-dimensional latent space in which high-frequency and imperceptible details are hidden.

[0083] In order to preprocess y of different modalities, a domain-specific encoder τθ is used, which can map y to an intermediate representation Then the intermediate representation is mapped to the middle layer of U-Net through the cross attention mechanism Attention(Q,K,V).

[0084] The noise map is first input into the U-Net network and converted into a hidden state through a series of convolutional layers and downsampling operations. This hidden state contains the high-level features of the noise map. In the decoding stage of U-Net, the cross-attention mechanism is introduced so that the network can not only pay attention to the features in its own hidden state, but also combine the description information and the defect mask design map to better generate the target image. Specifically, the cross-attention mechanism allows the network to consider the relevant features from the description information and the defect mask design map when generating each pixel, thereby ensuring that the generated image meets the given conditions. The U-Net network gradually restores the spatial resolution of the image through the decoder and calculates the predicted noise in the process. The predicted noise refers to the noise that the network estimates should be removed at the current time step. Finally, the predicted noise is used to denoise the noise map, gradually restore the original image or generate a new image. This process actually reverses the initial noise addition process, and finally generates the target image sample by gradually reducing the noise.

[0085] This embodiment converts the noise map into the hidden state of the U-Net network, adopts the cross-attention mechanism through the U-Net network, calculates the predicted noise based on the description information and the defect mask design drawing, and uses the predicted noise to denoise the noise map to obtain the target image sample. The U-Net network can generate high-quality output patterns while retaining the details of the input pattern, and can also more accurately locate and generate the position and morphology of the defect. Using the cross-attention mechanism, the U-Net network can better understand the relationship between the description information and the defect mask design drawing, thereby improving the model's adaptability to different defect types and scenarios. Taking the noise map, description information and defect mask design drawing as input, it can generate image samples with a high diversity.

[0086] In one embodiment, the noise map, description information and defect mask design map are input into the U-Net network, the noise map is converted into the hidden state of the U-Net network, and the U-Net network adopts a cross attention mechanism to calculate the formula for predicting noise based on the description information and the defect mask design map as shown in the following formula (1).

[0087]

[0088] where ∈θ(zt, t, τ θ(y)) represents the predicted noise, zt represents the hidden representation of the noise image at time step t, and τ θ(y) represents the intermediate representation of the encoded control condition input y. represents the hidden state of the U-Net network at layer i, Q represents the query vector, represents the query matrix, K represents the key vector, represents the weight matrix, K represents the value vector, represents the weight matrix, and DUNet represents the neural network based on the U-Net architecture.

[0089] The input of the U-Net model is the noise image zt, the time step t and the control condition y. In the embodiment of the present application, the control condition y is the text description information and the defect mask CAM map output is the predicted noise ∈θ(zt, t, τ θ(y)). First, the time step t is converted into a high-dimensional vector, and the control condition input y is encoded into an intermediate representation τ θ(y). The U-Net network extracts the features of the noise image zt. At the i-th layer, the hidden state is extracted. Compute query vectors via criss-cross attention mechanism Key Vector Sum value vector Calculating attention weights The hidden state of the U-Net network is The output of the attention mechanism is merged with Attention(Q, K, V). Through the decoder part of the U-Net network, the predicted noise ∈θ(zt, t, τ θ(y)) is output. UNe t represents a deep neural network based on the U-Net architecture.

[0090] The noise image is denoised using the predicted noise, and the formula for obtaining the target image sample is as follows: Formula (2).

[0091]

[0092] Among them, zt-1 represents the target image sample after denoising, zt represents the noise image at time step t, αt and βt represent the coefficients related to time step t, αt represents the proportion of image retention during forward diffusion, and βt represents the proportion of noise added during forward diffusion. represents the cumulative product of α values ​​at all previous time steps, and ∈θ represents the prediction noise.

[0093] Starting from the original image x, a series of noisy images zt are generated by gradually adding Gaussian noise, with each time step t corresponding to a noise level αt and βt. After predicting the noise ∈θ(zt, t, τ θ(y)) using the U-Net model. Calculate the denoising term represents subtracting the predicted noise ∈θ from the current noise image zt, Is a scaling factor used to adjust the strength of the predicted noise to ensure the stability of the denoising process. Subtract the denoising term from the current noisy image zt Multiply the denoised result by the scaling factor Used to adjust the strength of the retained image information. This factor ensures that enough image information is retained during the denoising process, rather than completely removing all noise, and the final denoised image is obtained

[0094] In one embodiment, a stable diffusion model, i.e., a defect image generation model, can be trained by collecting different OK images and NG images as training sample sets, and the diversity of defect shapes and positions can be ensured as much as possible during the collection. In the process of denoising the defect image generation model, the model is continuously optimized by minimizing the difference between the predicted noise and the actual noise, and the denoising performance is gradually improved during the training process. The loss function of the denoising diffusion model is formula (3).

[0095]

[0096] Among them, LLDM is the loss function, which represents the training target of the model. The loss function is used to measure the noise ∈θ(z t ,t,τθ(y)) and the difference between the actual noise∈; is the expected value, which means averaging over all possible inputs. represents the noise ∈θ(z t ,t,τθ(y)) and the actual noise∈.

[0097] This embodiment describes, through detailed formulas, how to use the U-Net network in combination with the cross-attention mechanism to predict noise and how to use the predicted noise to gradually reduce the noise map to generate a target image sample that is closer to the original image. Due to the use of multiple input forms such as noise maps, description information, and defect mask design maps, coupled with the powerful expression ability of the U-Net network, the model can more accurately estimate noise, improve the quality of image generation, and generate more diverse and realistic image samples.

[0098] Based on the same concept, the present application embodiment provides a graph sample generating device. Figure 4 The image sample generating device provided in the embodiment of the present application is described in detail.

[0099] Figure 4 It is a structural block diagram of a graph sample generating device shown in an embodiment of the present application.

[0100] like Figure 4 As shown, the image sample generating device may include:

[0101] The acquisition module is used to acquire image samples, description information of image samples, design drawings of image samples, and historical image samples obtained based on the design drawings. The historical image samples include defect categories, defect area information, and defect shapes.

[0102] The determination module is used to determine the target defect area information and target defect shape of the target defect type in the design drawing based on the historical drawing sample.

[0103] The adding module is used to add a defect mask to the design drawing according to the target defect area information and the target defect shape to obtain a defect mask design drawing.

[0104] The generation module is used to input the image sample, description information and defect mask design diagram into the image sample generation model, add noise to the image sample through the image sample generation model to obtain a noise image, and perform denoising on the noise image based on the noise image, description information and defect mask design diagram to obtain a target image sample.

[0105] In one embodiment, before obtaining the description information of the image sample, the acquisition module is also used to convert the image sample into an image embedding vector; calculate the similarity between the image embedding vector and the text embedding vector of the historical description information; and select the historical description information corresponding to the highest similarity as the description information of the image sample.

[0106] In one embodiment, the determination module is specifically used to determine all defect areas with a target defect type in the design image from the historical image samples; and determine a defect area that meets a preset condition from all defect areas as a target defect area.

[0107] In one embodiment, the determination module is specifically used to determine all defect shapes of the target defect type in the design drawing from the historical drawing samples; randomly select an original defect shape from all defect shapes, and sample the shape edge of the original defect shape to obtain sampling points; perform curve fitting based on the sampling points to obtain the target defect shape.

[0108] In one embodiment, the image sample generation model includes a U-Net network, and the generation model is specifically used to input the noise map, description information and defect mask design map into the U-Net network; convert the noise map into a hidden state of the U-Net network; use the cross-attention mechanism through the U-Net network to calculate the predicted noise based on the description information and the defect mask design map; use the predicted noise to denoise the noise map to obtain a target image sample.

[0109] In one embodiment, the generation model is specifically used to input the noise map, description information and defect mask design map into the U-Net network, convert the noise map into the hidden state of the U-Net network, and use the cross attention mechanism through the U-Net network to calculate the formula for predicting noise based on the description information and the defect mask design map, including:

[0110]

[0111] Among them, ∈θ(z t ,t,τθ(y)) represents the prediction noise, z t represents the hidden representation of the noise image at time step t, τθ(y) represents the intermediate representation of the encoded conditional input y, represents the hidden state of the U-Net network at layer i, Q represents the query vector, represents the query matrix, K represents the key vector, represents the weight matrix, K represents the value vector, represents the weight matrix, D UNet Represents a neural network based on the U-Net architecture;

[0112] The formula for denoising the noise image using the predicted noise to obtain the target image sample includes:

[0113]

[0114] Among them, z t-1 represents the target image sample after denoising, z t represents the noise image at time step t, α t and β t represents the coefficient associated with time step t, α t represents the proportion of image retention during the forward diffusion process, β trepresents the proportion of noise added during the forward diffusion process, represents the cumulative product of α values ​​at all previous time steps, and ∈θ represents the prediction noise.

[0115] In one embodiment of the present application, a defect recognition model can be trained based on the generated defect image samples. The defect recognition model adopts the SegFormer,architecture. The input image is converted into a fixed-length Token through Overlap Patch Embedding. Then, the feature extraction of 4 layers of Transformer Block is used to obtain multi-scale image features, and these features are then interactively fused through the MLP Layer to obtain a richer feature representation. Finally, a category distribution vector for each pixel is obtained through an MLP classification head. During training, the category distribution vector and the label are subjected to multi-category cross entropy calculation to obtain the loss, and then the loss is back-propagated and the model is learned. Finally, a defect recognition model is formed, and the defect recognition model trained by this model can effectively improve the accuracy of defect recognition.

[0116] Figure 4 Each module in the device shown has the function of realizing Figure 1 The functions of each step in the process can achieve the corresponding technical effects, which will not be described in detail here for the sake of brevity.

[0117] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application is shown.

[0118] The electronic device may include a processor 510 and a memory 520 storing computer program instructions.

[0119] Specifically, the processor 510 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0120] The memory 520 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 520 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 520 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 520 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 520 is a non-volatile solid-state memory.

[0121] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of the present disclosure.

[0122] The processor 510 implements any one of the image sample generation methods in the above embodiments by reading and executing computer program instructions stored in the memory 520 .

[0123] In one example, the electronic device may further include a communication interface 530 and a bus 540. Figure 5 As shown, the processor 510, the memory 520, and the communication interface 530 are connected via a bus 540 and communicate with each other.

[0124] The communication interface 530 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0125] Bus 540 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 540 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0126] The electronic device can execute the image sample generation method in the embodiment of the present application, thereby realizing the combination Figure 1 method.

[0127] In addition, in combination with the image sample generation method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any image sample generation method in the above embodiment is implemented.

[0128] An embodiment of the present application also provides a computer program product, including a computer program, which implements any one of the image sample generation methods in the above embodiments when the computer program is processed and executed.

[0129] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0130] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0131] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0132] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.

[0133] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for generating a graph sample, characterized in that: The method comprises: Acquire a picture sample, description information of the picture sample, a design drawing of the picture sample, and a historical picture sample obtained based on the design drawing, wherein the historical picture sample includes defect category, defect area information, and defect shape; Determine, based on the historical image sample, target defect region information and target defect shape of the target defect type in the design image; According to the target defect region information and the target defect shape, adding a defect mask to the design drawing to obtain a defect mask design drawing; The image sample, the description information and the defect mask design drawing are input into an image sample generation model, noise is added to the image sample through the image sample generation model to obtain a noise image, and noise reduction processing is performed on the noise image based on the noise image, the description information and the defect mask design drawing to obtain a target image sample.

2. The method according to claim 1, characterized in that Before obtaining the description information of the image sample, the method further includes: Converting the image sample into an image embedding vector; Calculating the similarity between the image embedding vector and the text embedding vector of the historical description information; The historical description information corresponding to the highest similarity is selected as the description information of the graph sample.

3. The method according to claim 1, characterized in that The determining, based on the historical image sample, a target defect area having a target defect type in the design image comprises: Determine from the historical image samples all defect areas in the design image that have a target defect type; A defect area that meets a preset condition is determined from among all the defect areas as the target defect area.

4. The method according to claim 1, characterized in that: The determining, based on the historical image sample, a target defect shape of a target defect type existing in the design image comprises: Determine all defect shapes of the target defect type in the design drawing from the historical drawing samples; Randomly selecting an original defect shape from all the defect shapes, and sampling the shape edge of the original defect shape to obtain sampling points; Curve fitting is performed based on the sampling points to obtain the target defect shape.

5. The method according to claim 1, characterized in that The image sample generation model includes a U-Net network, and the noise image is subjected to denoising based on the noise image, the description information and the defect mask design image to obtain a target image sample, including: Inputting the noise map, the description information and the defect mask design map into the U-Net network; Converting the noise map into a hidden state of the U-Net network; Using a cross attention mechanism through the U-Net network, the predicted noise is calculated based on the description information and the defect mask design diagram; The noise image is subjected to denoising processing using the predicted noise to obtain the target image sample.

6. The method according to claim 5, characterized in that The noise map, the description information and the defect mask design map are input into the U-Net network, the noise map is converted into a hidden state of the U-Net network, and a formula for calculating the predicted noise based on the description information and the defect mask design map by using a cross attention mechanism through the U-Net network includes: Among them, ∈ θ (z t ,t,τ θ (y)) represents the prediction noise, z t represents the hidden representation of the noisy image at time step t, τ θ (y) represents the intermediate representation of the encoded conditional input y, represents the hidden state of the U-Net network at layer i, Q represents the query vector, represents the query matrix, K represents the key vector, represents the weight matrix, K represents the value vector, represents the weight matrix, D UNet Represents a neural network based on the U-Net architecture; The formula for performing denoising on the noise image by using the predicted noise to obtain the target image sample includes: Among them, z t-1 represents the target image sample after denoising, z t represents the noise image at time step t, α t and β t represents the coefficient associated with time step t, α t represents the proportion of image retention during the forward diffusion process, β t represents the proportion of noise added during the forward diffusion process, represents the cumulative product of the α values ​​of all previous time steps, ∈ θ represents the prediction noise.

7. A device for generating a graph sample, characterized in that: The device comprises: An acquisition module, used to acquire a graph sample, description information of the graph sample, a design drawing of the graph sample, and a historical graph sample obtained based on the design drawing, wherein the historical graph sample includes a defect category, defect area information, and a defect shape; A determination module, configured to determine target defect region information and target defect shape of a target defect type in the design drawing based on the historical drawing sample; An adding module, configured to add a defect mask to the design drawing to obtain a defect mask design drawing according to the target defect area information and the target defect shape; A generation module is used to input the image sample, the description information and the defect mask design drawing into an image sample generation model, add noise to the image sample through the image sample generation model to obtain a noise image, and perform denoising on the noise image based on the noise image, the description information and the defect mask design drawing to obtain a target image sample.

8. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the image sample generation method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the image sample generation method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the image sample generation method as described in any one of claims 1 to 6.

Images