Defect image generation method and device, storage medium, and electronic device

Abstract

The application provides a defect image generation method and device, a storage medium and an electronic device, wherein the method comprises: acquiring a defect description text, a defect mask and a noise parameter, wherein the defect description text is used to represent the position, size and defect type of a defect image to be generated; a text encoder is used to encode the defect description text to obtain a defect feature vector, and a defect category generator is used to encode the defect feature vector to obtain a defect category parameter; and the defect image is generated according to the defect mask, the noise parameter, the defect feature vector and the defect category parameter. Through the embodiment of the application, the technical problem of unconvincing generation of a defect image in the related art is solved, and the problem of small samples and zero samples of an industrial defect detection large model is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial control, in particular to a defect image generation method and device, a storage medium and an electronic device. BACKGROUND

[0002] In the production process of automobile parts, a picture generation model based on stable diffusion is generally used, such as Controlnet or PowerPaint, DreamBooth.

[0003] The scheme of the related art has good effects when generating general objects such as flowers, books and animals because there are more reference images that can be searched, but when generating defect images for industrial automobile hub defect detection, the effects are not realistic enough, and the generated defect images are not different enough when switching between different defect categories.

[0004] No efficient and accurate solution has been found for the above problems in the related art. SUMMARY

[0005] The present application provides a defect image generation method and device, a storage medium and an electronic device to solve the above technical problems in the related art.

[0006] According to one embodiment of the present application, a defect image generation method is provided, comprising: obtaining a defect description text, a defect mask and a noise parameter, wherein the defect description text is used to represent the position, size and defect type of the defect image to be generated; using a text encoder to encode the defect description text to obtain a defect feature vector, and using a defect category generator to encode the defect feature vector to obtain a defect category parameter; generating the defect image according to the defect mask, the noise parameter, the defect feature vector and the defect category parameter.

[0007] Optionally, generating the defect image according to the defect mask, the noise parameter, the defect feature vector and the defect category parameter comprises: generating an original hidden vector using the defect mask, the noise parameter and the defect feature vector; generating a multi-modal information fused intermediate feature vector according to the original hidden vector; generating an optimized hidden vector using the defect category parameter and the intermediate feature vector; decoding the optimized hidden vector using a decoder to obtain the defect image.

[0008] Optionally, generating the intermediate feature vector after multi-modal information fusion according to the original latent vector comprises: inputting the original latent vector into a multi-head latent attention (MLA) layer, performing rank reduction conversion on a key-value (KV) matrix of the original latent vector through the MLA layer to obtain an attention matrix; performing layer normalization on the attention matrix to a standard normal distribution to obtain normalized data; inputting the normalized data into a feedforward network to obtain real vector data; performing residual connection on the attention matrix and the real vector data to obtain residual data; and performing linear transformation on the residual data to obtain the intermediate feature vector.

[0009] Optionally, encoding the defect feature vector by using a defect category generator to obtain a defect category parameter comprises: mapping the defect feature vector into a logit vector by using a fully connected neural network of the defect category generator, wherein a dimension of the logit vector is greater than a dimension of the defect feature vector, and the defect category generator comprises a fully connected neural network and a Softmax activation function; inputting the logit vector into the Softmax activation function to output a plurality of probability data; and selecting a target probability data with the highest value from the plurality of probability data, and outputting the target probability data as the defect category parameter.

[0010] Optionally, combining the defect category parameter and the intermediate feature vector to generate an optimized latent vector comprises: extracting an integer number in the defect category parameter; and combining the integer number and the intermediate feature vector to generate the optimized latent vector.

[0011] Optionally, decoding the optimized latent vector by using a decoder to obtain the defect image comprises: converting the optimized latent vector into an attention vector set by using the decoder, wherein the attention vector set comprises a key vector and a value vector, and the decoder comprises a self-attention layer; and decoding the attention vector set by using the self-attention layer to obtain the defect image.

[0012] Optionally, decoding the attention vector set by using the self-attention layer to obtain the defect image comprises: analyzing a sequence position of each feature vector in the attention vector set in the self-attention layer; selecting a plurality of feature vector elements with a sequence position less than a preset value in the attention vector set; judging whether the plurality of feature vector elements contain a preset end symbol; if the feature vector elements contain the preset end symbol, determining that the self-attention layer is decoded, and outputting the feature vector elements.

[0013] According to another embodiment of the present application, a device for generating a defect image is provided, comprising: an acquisition module configured to acquire a defect description text, a defect mask, and a noise parameter, wherein the defect description text is used to represent a position, a size, and a defect type of a defect image to be generated; an encoding module configured to encode the defect description text using a text encoder to obtain a defect feature vector, and encode the defect feature vector using a defect category generator to obtain a defect category parameter; and a generation module configured to generate the defect image according to the defect mask, the noise parameter, the defect feature vector, and the defect category parameter.

[0014] Optionally, the generation module comprises: a first generation unit configured to generate an original latent vector using the defect mask, the noise parameter, and the defect feature vector; a second generation unit configured to generate an intermediate feature vector after multi-modal information fusion according to the original latent vector; a third generation unit configured to generate an optimized latent vector by combining the defect category parameter and the intermediate feature vector; and a decoding unit configured to decode the optimized latent vector using a decoder to obtain the defect image.

[0015] Optionally, the second generation unit comprises: a conversion subunit configured to input the original latent vector into a multi-head latent attention (MLA) layer, and perform rank reduction conversion on a key-value (KV) matrix of the original latent vector through the MLA layer to obtain an attention matrix; a processing subunit configured to normalize the attention matrix to a standard normal distribution to obtain normalized data; a feedforward subunit configured to input the normalized data into a feedforward network to obtain real vector data; a connection subunit configured to perform residual connection on the attention matrix and the real vector data to obtain residual data; and a transformation subunit configured to perform linear transformation on the residual data to obtain the intermediate feature vector.

[0016] Optionally, the transformation subunit is further configured to: map the defect feature vector into a logit vector using a fully connected neural network of the defect category generator, wherein a scale of the logit vector is greater than a scale of the defect feature vector, and the defect category generator comprises a fully connected neural network and a Softmax activation function; input the logit vector into the Softmax activation function to output a plurality of probability data; and select a target probability data with the highest value from the plurality of probability data, and output the target probability data as the defect category parameter.

[0017] Optionally, the third generation unit comprises: an extraction subunit configured to extract an integer number in the defect category parameter; and a generation subunit configured to combine the integer number and the intermediate feature vector to generate the optimized latent vector.

[0018] Optionally, the decoding unit comprises a conversion subunit configured to convert the optimized latent vector into a set of attention vectors by using the decoder, wherein the set of attention vectors comprises key vectors and value vectors, and the decoder comprises a self-attention layer; and a decoding subunit configured to decode the set of attention vectors by using the self-attention layer to obtain the defect image.

[0019] Optionally, the decoding subunit is further configured to: parse the sequence position of each feature vector in the set of attention vectors in the self-attention layer; select a plurality of feature vector elements with a sequence position less than a preset value in the set of attention vectors; determine whether the preset end-of-sequence symbol is contained in the plurality of feature vector elements; and if the preset end-of-sequence symbol is contained in the plurality of feature vector elements, determine that the decoding of the self-attention layer is completed, and output the plurality of feature vector elements.

[0020] According to still another embodiment of the present application, a storage medium is provided, wherein the storage medium stores a computer program, and the computer program is configured to execute the steps in any of the above-mentioned device embodiments when running.

[0021] According to still another embodiment of the present application, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to execute the steps in any of the above-mentioned device embodiments.

[0022] By the embodiments of the present application, the defect description text, the defect mask, and the noise parameter are obtained, wherein the defect description text is used to represent the position, size, and defect type of the defect image to be generated; the defect description text is encoded by using the text encoder to obtain a defect feature vector, and the defect feature vector is encoded by using the defect category generator to obtain a defect category parameter; and the defect image is generated according to the defect mask, the noise parameter, the defect feature vector, and the defect category parameter, which realizes a user-defined text-to-image scheme, can generate images of multiple defect types with obvious differences, solves the technical problem of unconvincing defect images in the related art, and avoids the problems of small sample and zero sample. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are included to provide a further understanding of the present application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0024] Figure 1 is a hardware structure block diagram of a computer according to an embodiment of the present application;

[0025] Figure 2is a flow chart of a method for generating a defect image according to an embodiment of the present application;

[0026] Figure 3 is a flow chart of an embodiment of the present application;

[0027] Figure 4 is a slag inclusion defect map and a pore defect map generated by an embodiment of the present application;

[0028] Figure 5 is a structural block diagram of a device for generating a defect image according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable persons skilled in the art to better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0030] It should be noted that the terms “first”, “second”, and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms “include” and “have” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, product, or device including a series of steps or units does not necessarily have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product, or device.

[0031] Embodiment 1

[0032] The method embodiments provided in the embodiments of the present application can be executed in a server, a computer, a camera-like operation device. Taking the case of running on a computer as an example, Figure 1 is a hardware structural block diagram of a computer according to an embodiment of the present application. As shown in Figure 1 , the computer can include one or more ( Figure 1The computer shown in FIG. 1 includes only one processor 102 (the processor 102 can include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), and a memory 104 for storing data. Optionally, the computer can further include a transmission device 106 for communication functions, and an input / output device 108. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, and does not limit the structure of the computer. For example, the computer can include more or fewer components than those shown in FIG. 1, or have a different configuration from that shown in FIG. 1. Figure 1 Figure 1 The structure shown is only schematic, and does not limit the structure of the computer. For example, the computer can include more or fewer components than those shown in FIG. 1, or have a different configuration from that shown in FIG. 1.

[0033] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as a computer program corresponding to the defect image generation method of the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, implements the above-mentioned method. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, a flash memory, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, and these remote memories can be connected to the computer through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0034] The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network can include a wireless network provided by a communication provider of the computer. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC) which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module which is used to communicate with the Internet in a wireless manner.

[0035] In the present embodiment, a defect image generation method is provided, Figure 2 is a flowchart of a defect image generation method according to an embodiment of the present application, as shown in FIG. 2, the flow includes the following steps: Figure 2

[0036] In step S202, a defect description text, a defect mask, and a noise parameter are obtained, wherein the defect description text is used to represent the position, size, and defect type of the defect image to be generated;

[0037] ​​Optionally, the defect description text is a prompt, the defect mask is a single-channel picture matrix, and the noise parameter is a floating-point number.

[0038] Optionally, the defect image can be a hub defect image.

[0039] In step S204, the defect description text is encoded by using a text encoder to obtain a defect feature vector, and the defect feature vector is encoded by using a defect classification generator to obtain a defect classification parameter.

[0040] The encoder for the defect description text includes a text encoder and a defect classification generator. In a product for detecting defects in the industrial field, the categories of defects are limited, and a one-to-one correspondence can be established between the defect feature vector of the text and the defect category to be generated. The text encoder is responsible for encoding the description of the defect generation position and size in the prompt into a feature vector, and the defect classification generator is responsible for encoding the defect category corresponding to the defect feature vector into an integer number.

[0041] In step S206, the defect image is generated according to the defect mask, the noise parameter, the defect feature vector, and the defect classification parameter.

[0042] The generated defect image can be used to train an industrial defect detection large model for detecting product defects in this field.

[0043] Through the above steps, the defect description text, the defect mask, and the noise parameter are obtained, wherein the defect description text is used to represent the position, size, and defect type of the defect image to be generated. The defect feature vector is obtained by encoding the defect description text by using a text encoder, and the defect classification parameter is obtained by encoding the defect feature vector by using a defect classification generator. The defect image is generated according to the defect mask, the noise parameter, the defect feature vector, and the defect classification parameter. A user-defined text-to-image generation scheme is realized, which can generate images of multiple defect types with obvious differences, solves the technical problem of unrealistic generation of defect images in related technologies, and avoids the small sample and zero sample problems of the industrial defect detection large model.

[0044] In the embodiment, generating the defect image according to the defect mask, the noise parameter, the defect feature vector, and the defect category parameter comprises: generating a raw latent vector by using the defect mask, the noise parameter, and the defect feature vector; generating an intermediate feature vector after multi-modal information fusion according to the raw latent vector; generating an optimized latent vector by using the defect category parameter and the intermediate feature vector; and decoding the optimized latent vector by using a decoder to obtain the defect image.

[0045] Optionally, when the raw latent vector is generated by using a variational auto-encoder (VAE), the variational auto-encoder is responsible for combining the defect feature vector encoded by the text encoder, the embeddings of the defect mask encoded by the picture encoder, and the noise parameter, performing an exponential moving average (EMA) operation on the model input (the defect mask, the noise parameter, and the defect feature vector), averaging the parameters of the model to improve the performance and increase the robustness of the model, and further efficiently extracting data latent feature encoding, small defect features, and overall image texture to generate the raw latent vector.

[0046] In one embodiment, generating the intermediate feature vector after multi-modal information fusion according to the raw latent vector comprises: inputting the raw latent vector into a multi-head latent attention (MLA) layer, performing rank reduction conversion on a key-value (KV) matrix of the raw latent vector by using the MLA layer to obtain an attention matrix; performing layer normalization on the attention matrix to a standard normal distribution to obtain normalized data; inputting the normalized data into a feed forward network to obtain real vector data; performing residual connection on the attention matrix and the real vector data to obtain residual data; and performing linear transformation on the residual data to obtain the intermediate feature vector.

[0047] The feed forward network outputs a real vector data, residual connection is performed on the attention matrix and the real vector data to obtain a floating point number, and the floating point number is changed into a feature expression by linear transformation when the linear transformation layer performs linear transformation, and the intermediate feature vector after multi-modal information fusion is output.

[0048] In this embodiment, the MLA (Multi-Head Latent Attention) layer is used to greatly save the KV (K is Key and V is Value) cache, thereby significantly reducing the calculation cost. The MLA has loss compression for KV, improves the storage information density while retaining as much key details as possible, can significantly reduce the KV cache size, converts the KV matrix into a low-rank form, represents the original matrix as the product of two smaller matrices, and is equivalent to the product of latent vectors. In the inference process, only the latent vector is cached, and the complete key KV is not cached, which avoids the information loss of the grouped query attention and the multi-query attention query, thereby obtaining better performance under the premise of reducing the KV cache.

[0049] In this embodiment, the layer normalization normalizes the hidden layer in the neural network to a standard normal distribution, that is, independent and identically distributed, so as to accelerate the training speed and accelerate the convergence:

[0050]

[0051] In the above formula, the mean is calculated in the row of the matrix row;

[0052]

[0053] In the above formula, the variance is calculated in the row of the matrix row;

[0054] The calculation formula of the layer normalization is:

[0055]

[0056] wherein, is the input data, m is the number of j-dimensional input data, is the mean, is the standard deviation, then each element in each row is subtracted by the mean of the row and divided by the standard deviation of the row, thereby obtaining the normalized value, is to prevent division by 0, and then two trainable parameters are introduced to make up for the information lost in the normalization process, represents element multiplication, and the initialization is all 1, is all 0.

[0057] In the residual connection, the V after the attention matrix weighting, that is, , is transposed to make it consistent with the dimension of the input (original hidden vector) , that is, Then we add them up to do the residual connection, add the corresponding elements because they have the same dimension:

[0058]

[0059] In the later operation, after each module operation, the value before the operation and the value after the operation are added to obtain the residual connection, so that the gradient can be directly transmitted to the initial layer during training:

[0060] .

[0061] In one example, the defect feature vector is encoded by using a defect category generator to obtain a defect category parameter, including: using a fully connected neural network of the defect category generator to map the defect feature vector to a logit vector, wherein the dimension of the logit vector is larger than the dimension of the defect feature vector, and the defect category generator includes a fully connected neural network and a Softmax activation function; inputting the logit vector into the Softmax activation function to output a plurality of probability data; selecting a target probability data with the highest value from the plurality of probability data, and outputting the target probability data as the defect category parameter.

[0062] The defect category generator includes a fully connected neural network and a Softmax activation function, which can project the vector generated by the text encoder into a much larger vector called logit (logit vector), and the following Softmax converts the scores into probabilities (all positive numbers, upper limit 1.0). The target probability data with the highest probability is selected, and the defect category corresponding to it is taken as the output of this step, i.e. the defect category parameter.

[0063] In one embodiment, combining the defect category parameter and the intermediate feature vector to generate an optimized latent vector includes: extracting an integer number in the defect category parameter; and combining the integer number and the intermediate feature vector to generate an optimized latent vector.

[0064] The output result (intermediate feature vector) of the Linear Layer and the integer number of the defect category generated by the defect category generator are combined into a Refined Latent (optimized latent vector), which contains all the input information high-dimensional features.

[0065] In an embodiment, decoding the optimized latent vector to obtain the defect image comprises: converting the optimized latent vector into an attention vector set by using the decoder, wherein the attention vector set comprises key vectors and value vectors, and the decoder comprises a self-attention layer; and decoding the attention vector set by using the self-attention layer to obtain the defect image.

[0066] In the embodiment, the VAE decoder reconstructs the latent feature (optimized latent vector) into a pixel-level image, and the VAE (variational autoencoder) additionally comprises an Encoder-Decoder Attention. The Encoder-Decoder Attention works in the same way as the multi-head self-attention layer, that is, a Query matrix is created through the layer below it, and a key / value matrix is obtained from the output of the encoder.

[0067] In an example, decoding the attention vector set by using the self-attention layer to obtain the defect image comprises: analyzing the sequence position of each feature vector in the attention vector set in the self-attention layer; selecting a plurality of feature vector elements with a sequence position less than a preset value in the attention vector set; determining whether the plurality of feature vector elements contain a preset end symbol; and if the feature vector elements contain the preset end symbol, determining that the decoding of the self-attention layer is completed, and outputting the feature vector elements.

[0068] The VAE decoder starts to work by processing the input optimized latent vector, and the output of the top encoder is converted into an attention vector set comprising vectors K (key vector) and V (value vector). The matrix will be used by the decoder for the Encoder-Decoder Attention layer of the decoder, which can help the decoder to focus on the specific position of the input sequence.

[0069] The self-attention layer (Encoder-Decoder Attention layer) in the VAE decoder is only allowed to process the earlier positions in the output sequence in the decoder. Before the Softmax step, it will hide the later positions by setting them to -inf. Like the setting for the input of the encoder, the position encoding is embedded and added to the decoder to represent the position of each feature vector. Each sub-step in the decoding stage outputs an element of the output sequence until a special end symbol is reached, which indicates that the decoder has completed the decoding.

[0070] The VAE decoder of the embodiment is implemented by a neural network, denoted as where θ represents the parameters of the network. The task of the decoder is to accept the latent variable z as input (optimized latent vector) and generate a reconstructed data (defect image):

[0071] =

[0072] wherein, The reconstructed image or data can be regarded as a function of the neural network parameterized by θ. The decoder defines a conditional distribution , representing the generation distribution of data x given the latent variable z, assuming that ) is a Gaussian distribution, in the form of:

[0073] =N( , I)

[0074] wherein, is the mean of the conditional distribution, i.e., the reconstruction result predicted by the neural network, I is the covariance matrix, is a hyperparameter representing the variance, and I is the identity matrix, with each dimension being independent.

[0075] In actual production, the actual hub production line generally has a high yield, and some defects, especially defect images of specific positions, specific sizes, are difficult to collect, and even occasionally new defects that cannot collect samples may occur. This type of defect is prone to cause batch missed detection. The present embodiment is based on a Diffusion Transformer (DiT) noise diffusion model to generate images from text, using image generation technology, to customize the generation of defect images of specified positions, specified categories and sizes on the hub according to user needs, which can be applied to the production of automobile hubs in the industrial field. Input the textual description of the defect image to be generated and the mask of the pre-generated defect position and size, generate a large number of industrial defect detection pictures, and use these generated pictures to train an AI large model for quality defect detection in the production process of automobile hubs. Figure 3is the overall flowchart of the embodiment of the present application, the input includes the Prompt to be generated defect image, the mask, the noise, the Clip Text Encoder (Clip ((Contrastive Language-Image Pre-training) text encoder) includes a text encoder (Text Encoder) and a defect classification generator (Defect Classification Generator), the data flow is sequentially through a variational auto-encoder (VAE, Variational Auto-Encoder), a raw latent (Raw Latent) is generated, a multi-head latent attention (MLA, Multi-Head Latent Attention), a layer normalization (Layer Normalization), a feed forward (Feed Forward), a linear layer (Linear Layer), a refined latent (Refined Latent) is generated, a VAE decoder, and finally a defect image is generated.

[0076] The present application is based on multi-modal visual language model (VLMs) and DiT (Diffusion Transformer) to generate pictures from text and generate pictures from pictures technology, in the training process, the network gradually learns to generate defect simulation pictures according to the input Prompt and mask, compared with the general generation model, the model proposed in the present application is more likely to generate diverse and realistic industrial field defect images, and the distance between different categories of defect features is large.

[0077] In the field of automobile parts defect detection, the embodiment can generate a large number of user-defined defect images according to the user's Prompt, i.e. the prompt engineering of AIGC (artificial intelligence generated content), the text description of the defect image to be generated and the mask of the pre-generated defect position and size. Figure 4 is the slag inclusion defect image and the porosity defect image generated by the embodiment of the present application, these generated pictures can be used to train the AI detection large model of the defects generated in the production process of the product.

[0078] By adopting the scheme of the embodiment, in the early stage of project development, the effect that can be finally reached by the project can be quickly evaluated, the trial and error cost is reduced, and the overall input-output ratio is improved. The defect generation scheme described in the present application generates defect data for small sample, high yield, new defect missed detection, etc., solves the problems of small sample, zero sample, etc., and can be quickly (reduce 60% cycle) deployed and implemented.

[0079] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software on a general hardware platform as necessary, and of course can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a part that contributes to the prior art, and the computer software product is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk), and includes a plurality of instructions for causing an end device (which can be a mobile phone, a computer, a server, or a network device) to execute the method described in each embodiment of the present application.

[0080] Embodiment 2

[0081] In this embodiment, a defect image generation apparatus is also provided, which is used to implement the above embodiments and preferred embodiments, and will not be described again. The term "module" as used below can be a combination of software and hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementation of hardware or a combination of software and hardware can also be conceived.

[0082] Figure 5 is a structural block diagram of a defect image generation apparatus according to an embodiment of the present application, as shown in Figure 5 The apparatus includes:

[0083] The acquisition module 50 is configured to acquire a defect description text, a defect mask, and a noise parameter, wherein the defect description text is used to represent a position, a size, and a defect type of a defect image to be generated.

[0084] The encoding module 52 is configured to encode the defect description text by using a text encoder to obtain a defect feature vector, and encode the defect feature vector by using a defect category generator to obtain a defect category parameter.

[0085] The generation module 54 is configured to generate the defect image according to the defect mask, the noise parameter, the defect feature vector, and the defect category parameter.

[0086] Optionally, the generation module includes: a first generation unit configured to generate an original latent vector by using the defect mask, the noise parameter, and the defect feature vector; a second generation unit configured to generate a multi-modal information fused intermediate feature vector according to the original latent vector; a third generation unit configured to generate an optimized latent vector by combining the defect category parameter and the intermediate feature vector; and a decoding unit configured to decode the optimized latent vector by using a decoder to obtain the defect image.

[0087] Optionally, the second generating unit comprises: a conversion subunit, configured to input the original hidden vector into a multi-head latent attention (MLA) layer, and perform rank reduction conversion on a key-value (KV) matrix of the original hidden vector through the MLA layer to obtain an attention matrix; a processing subunit, configured to perform layer normalization on the attention matrix to a standard normal distribution to obtain normalized data; a feedforward subunit, configured to input the normalized data into a feedforward network to obtain real vector data; a connection subunit, configured to perform residual connection on the attention matrix and the real vector data to obtain residual data; and a transformation subunit, configured to perform linear transformation on the residual data to obtain an intermediate feature vector.

[0088] Optionally, the transformation subunit is further configured to: map the defect feature vector into a logit vector by using a fully connected neural network of the defect category generator, where a dimension of the logit vector is greater than a dimension of the defect feature vector, and the defect category generator comprises a fully connected neural network and a Softmax activation function; input the logit vector into the Softmax activation function to output a plurality of probability data; and select a target probability data with the highest value from the plurality of probability data, and output the target probability data as a defect category parameter.

[0089] Optionally, the third generating unit comprises: an extraction subunit, configured to extract an integer number in the defect category parameter; and a generation subunit, configured to combine the integer number and the intermediate feature vector to generate an optimized hidden vector.

[0090] Optionally, the decoding unit comprises: a conversion subunit, configured to convert the optimized hidden vector into an attention vector set by using a decoder, where the attention vector set comprises a key vector and a value vector, and the decoder comprises a self-attention layer; and a decoding subunit, configured to decode the attention vector set by using the self-attention layer to obtain the defect image.

[0091] Optionally, the decoding subunit is further configured to: analyze a sequence position of each feature vector in the attention vector set in the self-attention layer; select a plurality of feature vector elements with a sequence position less than a preset value in the attention vector set; determine whether the plurality of feature vector elements contain a preset end-of-sequence symbol; and if the plurality of feature vector elements contain the preset end-of-sequence symbol, determine that the self-attention layer is decoded and output the plurality of feature vector elements.

[0092] It should be noted that the above modules can be implemented by software or hardware, and for the latter, the following implementation manners can be used, but are not limited thereto: all the modules are located in the same processor; or the modules are located in different processors in any combination.

[0093] Embodiment 3

[0094] The embodiment of the present application also provides a storage medium, wherein the storage medium stores a computer program, and the computer program is configured to execute the steps in any of the method embodiments when running.

[0095] Optionally, in the embodiment, the storage medium is configured to store the computer program for execution.

[0096] S1, acquire a defect description text, a defect mask and a noise parameter, wherein the defect description text is used to represent a position, a size and a defect type of a defect image to be generated;

[0097] S2, encode the defect description text by using a text encoder to obtain a defect feature vector, and encode the defect feature vector by using a defect category generator to obtain a defect category parameter;

[0098] S3, generate the defect image according to the defect mask, the noise parameter, the defect feature vector and the defect category parameter.

[0099] Optionally, in the embodiment, the storage medium can include but is not limited to a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk and various storage media that can store a computer program.

[0100] The embodiment of the present application also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to execute the steps in any of the method embodiments.

[0101] Optionally, the electronic device can further comprise a transmission device and an input and output device, wherein the transmission device is connected with the processor, and the input and output device is connected with the processor.

[0102] Optionally, in the embodiment, the processor can be configured to execute the following steps by using the computer program.

[0103] S1, acquire a defect description text, a defect mask and a noise parameter, wherein the defect description text is used to represent a position, a size and a defect type of a defect image to be generated;

[0104] S2, encode the defect description text by using a text encoder to obtain a defect feature vector, and encode the defect feature vector by using a defect category generator to obtain a defect category parameter;

[0105] S3, generating the defect image according to the defect mask, the noise parameter, the defect feature vector, and the defect category parameter.

[0106] Optionally, specific examples in the embodiments can refer to the examples described in the above embodiments and optional implementation manners, and the embodiments will not be described here again.

[0107] The sequence numbers of the above-described embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0108] In the above-described embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0109] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the unit embodiment described above is only schematic. For example, the division of the units is only a logical function division. There can be another division manner for actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between the units can be indirect coupling or communication connection through some interface, electrical or other form.

[0110] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they can be located in one place, or they can be distributed on a plurality of network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiments.

[0111] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware, or in the form of a software functional unit.

[0112] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0113] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A method of generating a defect image, characterized by, The method comprises: obtaining defect description text, a defect mask, and noise parameters, wherein the defect description text is used to represent the position, size, and defect type of a defect image to be generated; encoding the defect description text using a text encoder to obtain a defect feature vector, and encoding the defect feature vector using a defect category generator to obtain defect category parameters; generating the defect image according to the defect mask, the noise parameters, the defect feature vector, and the defect category parameters; wherein generating the defect image according to the defect mask, the noise parameters, the defect feature vector, and the defect category parameters comprises: generating an original hidden vector using the defect mask, the noise parameters, and the defect feature vector; generating an intermediate feature vector after multi-modal information fusion according to the original hidden vector; generating an optimized hidden vector by combining the defect category parameters and the intermediate feature vector; and decoding the optimized hidden vector using a decoder to obtain the defect image; wherein encoding the defect feature vector using the defect category generator to obtain defect category parameters comprises: mapping the defect feature vector to a logit vector using a fully connected neural network of the defect category generator, wherein the scale of the logit vector is greater than the scale of the defect feature vector, and the defect category generator comprises a fully connected neural network and a Softmax activation function; inputting the logit vector into the Softmax activation function to output a plurality of probability data; and selecting a target probability data with the highest value from the plurality of probability data, and outputting the target probability data as the defect category parameters.

2. The method of claim 1, wherein, Generating an intermediate feature vector after multi-modal information fusion according to the original hidden vector comprises: inputting the original hidden vector into a multi-head latent attention (MLA) layer, performing rank reduction conversion on a key-value (KV) matrix of the original hidden vector through the MLA layer to obtain an attention matrix; performing layer normalization on the attention matrix to a standard normal distribution to obtain normalized data; inputting the normalized data into a feedforward network to obtain real vector data; performing residual connection on the attention matrix and the real vector data to obtain residual data; performing linear transformation on the residual data to obtain an intermediate feature vector.

3. The method of claim 1, wherein, Generating an optimized hidden vector by combining the defect category parameters and the intermediate feature vector comprises: extracting an integer number in the defect category parameters; combining the integer number and the intermediate feature vector to generate an optimized hidden vector.

4. The method of claim 1, wherein, Decoding the optimized hidden vector using a decoder to obtain the defect image comprises: converting the optimized hidden vector into an attention vector set using the decoder, wherein the attention vector set includes a key vector and a value vector, and the decoder includes a self-attention layer; decoding the attention vector set using the self-attention layer to obtain the defect image.

5. The method of claim 4, wherein, Decoding the attention vector set using the self-attention layer to obtain the defect image comprises: analyzing the sequence position of each feature vector in the attention vector set in the self-attention layer; selecting a number of feature vector elements whose sequence position in the attention vector set is less than a preset value; determining whether the number of feature vector elements contains a preset termination symbol; if the number of feature vector elements contains the preset termination symbol, determining that the self-attention layer decoding is completed, and outputting the feature vector element.

6. An apparatus for generating a defect image, characterized by comprising: comprising: an acquisition module configured to acquire a defect description text, a defect mask, and a noise parameter, wherein the defect description text is used to represent a position, a size, and a defect type of a defect image to be generated; an encoding module configured to encode the defect description text using a text encoder to obtain a defect feature vector, and encode the defect feature vector using a defect category generator to obtain a defect category parameter; a generation module configured to generate the defect image according to the defect mask, the noise parameter, the defect feature vector, and the defect category parameter; wherein the generation module comprises: a first generation unit configured to generate an original latent vector using the defect mask, the noise parameter, and the defect feature vector; a second generation unit configured to generate an intermediate feature vector fused with multi-modal information according to the original latent vector; a third generation unit configured to generate an optimized latent vector by combining the defect category parameter and the intermediate feature vector; and a decoding unit configured to decode the optimized latent vector using a decoder to obtain the defect image; wherein the encoding module is further configured to: map the defect feature vector to a logit vector using a fully connected neural network of the defect category generator, wherein a scale of the logit vector is greater than a scale of the defect feature vector, and the defect category generator comprises a fully connected neural network and a Softmax activation function; input the logit vector into the Softmax activation function to output a plurality of probability data; and select a target probability data with the highest value from the plurality of probability data, and output the target probability data as the defect category parameter.

7. A storage medium, characterized by A storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 5 when running. 8.An electronic device comprising a memory and a processor, the electronic device comprising: A memory stores a computer program, and a processor is configured to execute the computer program to execute the method described in any one of claims 1 to 5.

Images