Defect image generation method, apparatus and device

Abstract

The application relates to a defect image generation method, device and equipment, which comprises the following steps: loading a target model after training; the target model comprises a defect image generation model; the defect image generation model comprises a condition control network and a diffusion model; inputting target process parameters, a basic object image of a target object without defects and a preset defect description text into a feature extraction network of the diffusion model to perform feature coding and obtaining target features; inputting a preset defect mask image into the condition control network to obtain defect control features; and performing decoding processing on the target features and the defect control features based on a decoder of the diffusion model to generate a defect image of the target object. The method can quickly and accurately obtain a defect image.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus and device for generating defective images. Background Technology

[0002] In the field of computer vision technology, scenarios often require a large number of defect images. For example, in automated defect detection, to achieve good results, a large number of defect images are typically used for deep learning to train the defect detection model. More specifically, in the field of semiconductor testing technology, the identification of defects such as polishing debris, organic residues (e.g., photoresist, solvents), scratches, fractures, and bubbles from the mechanical grinding process on wafer surfaces is crucial, and typically requires a large number of wafer defect images to train the wafer defect detection model.

[0003] Relying solely on manual collection or photography of defect images presents the problems of high cost and low efficiency. Therefore, it is crucial to automatically generate reasonable and accurate defect images. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for generating defective images to address the aforementioned technical problems.

[0005] Firstly, a method for generating defect images is provided, the method comprising:

[0006] Load the trained target model; the target model includes a defect image generation model; the defect image generation model includes a conditional control network and a diffusion model.

[0007] The target process parameters, the base object diagram of the target object without defects, and the preset defect description text are input into the feature extraction network of the diffusion model for feature encoding to obtain the target features;

[0008] Input the preset defect mask image into the conditional control network to obtain the defect control features;

[0009] The decoder of the diffusion model performs decoding processing based on target features and defect control features to generate a defect image of the target object.

[0010] In the above method, a unique target model is pre-trained. The defect image generation model in this target model can combine the features of process parameters, image features, and defect control features to generate defect images that conform to the process parameters. In this way, the generated defect images not only meet the requirements in terms of morphology, but also more closely resemble the defect state in the real production environment, thus enabling the rapid and accurate acquisition of defect images.

[0011] In one embodiment, the target model further includes a process parameter prediction model; the target process parameter is the input process parameter; the method further includes: predicting the process parameter corresponding to the first target image through the process parameter prediction model; the first target image includes a base object image or a defect image; and generating a prompt message when the difference between the process parameter predicted for the first target image and the target process parameter exceeds a preset deviation range.

[0012] In the above scheme, the status of the equipment can be automatically monitored using a process parameter prediction model. That is, the process parameters predicted by the process parameter prediction model are compared with the actual input process parameters. If the difference is too large, an alarm message can be automatically output.

[0013] In one embodiment, the training process of the target model includes a first training phase for training a process parameter prediction model and a second training phase for training a defect image generation model. The training steps of the target model include:

[0014] In the first training phase, the process parameter prediction model is trained only using a first defect sample image that carries process parameter labels but does not carry noise.

[0015] The trained process parameter prediction model is used to predict the corresponding first process parameter for the second defect sample image with missing process parameters and no noise. The first process parameter is used as the process parameter label of the second defect sample image. A first sample set with noise is formed based on the first defect sample image and the second defect sample image.

[0016] In the second training phase, the pre-trained diffusion model is locked and not trained. The conditional control network is trained using the defect sample images and corresponding first defect annotation information from the first sample set to obtain the trained defect image generation model. The first defect annotation information includes the process parameter labels, defect text annotations, and defect pixel-level annotations corresponding to the defect sample images.

[0017] In the above scheme, the target model is trained in stages. First, a small number of defect sample images carrying process parameter labels but without noise are used to train the process parameter prediction model. Then, the trained process parameter prediction model is used to automatically and accurately predict and complete the process parameters that match the product appearance for defect sample images lacking process parameter labels. Based on the predicted process parameters as process parameter labels, the defect sample images are effectively expanded, improving expansion efficiency and reducing sample costs. Finally, a large number of expanded and completed defect sample images carrying process parameter labels are used to train the defect image generation model, thereby effectively improving the accuracy of the defect image generation model.

[0018] In one embodiment, the training steps of the target model include:

[0019] Obtain a second sample set containing a third defect sample image carrying second defect annotation information and noise; the second defect annotation information includes process parameter labels, defect text annotations, and defect pixel-level annotations corresponding to the third defect sample image;

[0020] In each round of model training, based on the defect image generation model to be trained in this round, a first defect image corresponding to each third defect sample image is generated;

[0021] Based on the process parameter prediction model to be trained in this round, predict the second process parameters corresponding to each third defect sample image;

[0022] Based on the noise loss and parameter loss corresponding to each third defect sample image, the corresponding target loss is obtained; the noise loss represents the difference between the noise predicted when generating the first defect image and the noise carried by the third defect sample image, and the parameter loss represents the difference between the second process parameter and the process parameter label corresponding to the third defect sample image.

[0023] Calculate the total loss based on the target loss corresponding to each third defect sample image;

[0024] Based on the total loss, the conditional control network in the process parameter prediction model and defect image generation model to be trained in this round is adjusted, and then the model is trained in the next round until the target model is obtained.

[0025] In the above scheme, the process parameter prediction model is implicitly trained while serving the generation task, and the defect image generation model also obtains better conditional input due to the more reasonable prediction of process parameters. The two evolve together to achieve better end-to-end training, thereby improving the accuracy of the defect image generation model and the process parameter prediction model, and subsequently generating defect images more accurately.

[0026] In one embodiment, the method further includes, in each round of model training:

[0027] Using the second process parameter as the new process parameter label, the defect image generation model is used to regenerate the second defect image corresponding to each third defect sample image based on the updated process parameter label.

[0028] Based on the feature differences between the second and first defect images corresponding to each third defect sample image, the reconstruction feature loss is obtained;

[0029] Based on the noise loss and parameter loss corresponding to each third defect sample image, the corresponding target loss is obtained as follows:

[0030] Based on the noise loss, parameter loss, and reconstruction feature loss corresponding to each third defect sample image, the corresponding target loss is obtained.

[0031] In the above scheme, for sample data with process parameter labels, such as the third defect sample image, the reconstruction feature loss can indirectly constrain the predicted second process parameter to approximate the true process parameter label. Therefore, combining noise loss, parameter loss, and reconstruction feature loss can determine a more accurate target loss, thereby training the model more effectively and quickly.

[0032] In one embodiment, the second sample set further includes a fourth defect sample image carrying third defect annotation information and noise; the third defect annotation information includes defect text annotations and defect pixel-level annotations, but does not include process parameter labels; in each round of model training, a corresponding first defect image, a second defect image, and a second process parameter are also generated for the fourth defect sample image. The method further includes:

[0033] In each round of model training, based on the process parameter prediction model to be trained in this round, the third process parameter is predicted for the second defect image corresponding to the fourth defect sample image.

[0034] The predicted stability characterization loss is determined based on the difference between the third process parameter and the second process parameter corresponding to each fourth defect sample image.

[0035] Based on the noise loss and prediction stability characterization loss corresponding to each fourth defect sample image, the corresponding target loss is obtained;

[0036] Based on the target loss corresponding to each third defect sample image, the total loss is calculated, including:

[0037] Calculate the total loss based on the target loss corresponding to each third defect sample image and each fourth defect sample image.

[0038] In the above scheme, for sample data without process parameter labels, such as the fourth defect sample image, the prediction stability characterization loss can represent the prediction stability of the process parameter prediction model. Combining the noise loss and the prediction stability characterization loss to determine the target loss for model training can improve the stability and rationality of the process parameter prediction model in the case of no process parameter labels. Furthermore, the more reasonable predicted process parameters enable the defect image generation model to be trained better, effectively improving the accuracy of the defect image generation model.

[0039] In one embodiment, the target model further includes a process parameter prediction model; the process parameter prediction model includes a feature extraction network and a parameter prediction module shared with the diffusion model;

[0040] Before inputting the target process parameters, the base object diagram of the target object without defects, and the pre-defined defect description text into the feature extraction network of the diffusion model for feature encoding to obtain the target features, the method also includes:

[0041] The second target image is input into the feature extraction network of the process parameter prediction model to obtain image features; the second target image includes a base object image or a defect reference image.

[0042] The parameter prediction module predicts the process parameters corresponding to the second target image based on image features.

[0043] The process parameters predicted for the second target image are used as the target process parameters.

[0044] In the above solution, manual input from the user is unnecessary. Instead, the target process parameters for generating defect images are directly predicted based on the process parameter prediction model, saving on manual operation costs. Furthermore, combining parameter prediction and image generation significantly improves efficiency.

[0045] Secondly, this application also provides a defect image generation apparatus, the apparatus comprising:

[0046] The model determination unit is used to load the trained target model; the target model includes a defect image generation model; the defect image generation model includes a conditional control network and a diffusion model.

[0047] The feature extraction unit is used to input the target process parameters, the basic object diagram of the target object without defects, and the preset defect description text into the feature extraction network of the diffusion model for feature encoding to obtain the target features;

[0048] The condition control unit is used to input a preset defect mask image into the condition control network to obtain defect control features;

[0049] The image reconstruction unit is used to generate a defect image of the target object by decoding based on the target features and defect control features through the decoder of the diffusion model.

[0050] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described in the first aspect.

[0051] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0052] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating a defect image generation method in one embodiment;

[0054] Figure 2 This is a schematic diagram of the framework of a defect image generation model in one embodiment;

[0055] Figure 3 This is a schematic diagram of the framework of a process parameter prediction model in one embodiment;

[0056] Figure 4 This is a schematic diagram of the framework of the target model in one embodiment;

[0057] Figure 5 This is a structural block diagram of a defect image generation device in one embodiment;

[0058] Figure 6 This is a structural block diagram of a defect image generation device in another embodiment;

[0059] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by those skilled in the art. The technical terms used herein are for the purpose of describing specific examples and are not intended to limit the scope of protection of this application.

[0061] Some approaches linearly increase the number of defective images by performing basic geometric transformations such as rotation, scaling, flipping, copying, and simple noise superposition on existing defective images. This approach tends to limit sample diversity, and the geometric transformations can easily corrupt semantics, generating erroneous defective images that contradict reality.

[0062] Some solutions employ Generative Adversarial Networks (GANs), utilizing a game-like process between the generator and discriminator to attempt to generate defect images from scratch. While this approach can overcome the aforementioned problems associated with geometric transformations, it suffers from insufficient physical realism. This is because image generation models trained on GANs learn only from image features (i.e., in deep learning, they only learn features specific to the content of sample images). Thus, the image generation model can only mimic the appearance of defects, but not their origin. This results in visually random defects that fail to reflect the actual physical causes on the production line, leading to inaccurate defect images that differ from those in real-world production environments.

[0063] Based on this, a defect image generation method is proposed, which can achieve deep cross-domain fusion of process parameters in real production environment and AI image generation algorithm, so that the generated defect distribution can change with the process parameters (e.g., exposure dose, deposition rate and other process parameters in chip production stage), thereby generating defect images that are closer to the real production environment.

[0064] It should be noted that the defect image generation method proposed in this application is applicable to any field or scenario that requires the generation of defect images using process parameters, and is not limited to scenarios involving the generation of defective wafer images in the field of chip technology. The following section will provide a more detailed description of this defect image generation method.

[0065] like Figure 1 As shown, in some embodiments, a defect image generation method is provided. This method is executed by a computer device, which is a testing machine or other electronic device with defect image generation capabilities. The method specifically includes the following steps:

[0066] S11, Load the trained target model; the target model includes a defect image generation model; the defect image generation model includes a conditional control network and a diffusion model.

[0067] In the actual inference application of the trained target model (i.e., the use case or inference stage of the target model), the target model can be loaded. The target model is a deep learning model that implements the defect image generation method in the various embodiments of this application. The target model includes a defect image generation model, which is used to generate defect images.

[0068] For example, the target model may be just a defect image generation model, or it may include other deep learning models besides the defect image generation model, such as a process parameter prediction model, which can predict the corresponding process parameters based on the image (i.e., predict the process parameters in the production environment that generated the image).

[0069] The defect image generation model in this application differs from traditional image generation models in at least the following ways:

[0070] (1) The defect sample images used to train the defect image generation model have corresponding process parameter labels. In other words, in this embodiment of the application, in addition to the image features of the defect sample images themselves, process parameters are also considered during model training. Thus, the trained defect image generation model can generate defect images that conform to the process parameters in the real production environment.

[0071] (2) The defective image generation model is trained based on the Conditional Control Diffusion Model (ControlNet) framework, which is different from the traditional image generation model trained based on the Generative Adversarial Network framework in terms of model architecture.

[0072] Specifically, the defect image generation model includes a conditional control network and a diffusion model. The diffusion model itself does not interpret the input control information or conditions; its main role is to provide the fundamental capabilities and rich details for image generation. The conditional control network is primarily used to inject external control conditions specific to the defect into the defect image generation process.

[0073] In summary, this application proposes a surface defect image synthesis technique that combines process parameters in the production environment with a condition-controlled diffusion model (ControlNet). Based on this technique, a defect image generation model is trained. Subsequent use of this model can generate higher quality defect images that more accurately match the process parameters in the real production environment.

[0074] S12, the target process parameters, the basic object diagram of the target object without defects, and the preset defect description text are input into the feature extraction network of the diffusion model for feature encoding to obtain the target features.

[0075] S13, input the preset defect mask image into the conditional control network to obtain the defect control features.

[0076] It should be understood that the input information of the target model includes target process parameters, a base object graph of the target object without defects, a preset defect description text, and a preset defect mask image. Specifically, the target process parameters, the base object graph of the target object without defects, and the preset defect description text are input into the diffusion model of the defect image generation model; the defect mask image is input into the conditional control network of the defect image generation model.

[0077] Here, the base object image refers to a normal surface image of the target object without defects. The base object image is equivalent to a background image; subsequently, defects need to be added to the base object image as a background to generate a defect image. Taking a wafer as an example, the image acquired from the surface of the wafer without defects is the base object image. In the embodiments of this application, defects can be added to the wafer in the base object image through a defect image generation model to generate a defect image.

[0078] Target process parameters are the process parameters used to guide the generation of defect images. Process parameters refer to the production parameters used in the manufacturing process of the target object. Taking a wafer as an example, the production parameters used when processing and manufacturing a wafer are the process parameters, including but not limited to the exposure dose, photoresist thickness, development time, and post-baking temperature in the photolithography stage; and the blade thickness, rotation speed, and feed rate in the dicing stage.

[0079] The defect image of the target object generated in step S14 conforms to the target process parameters. In other words, the generated defect image is very close to the real defect image under the target process parameters (i.e., the defect image generated or collected when using the target process parameters in a real production environment).

[0080] The defect description text is used to describe the defect features specified by the user (such as "irregular shape, dark gray, blurred edges," etc.). The defect description text guides the defect image generation model to generate defect images that conform to the specified defect features.

[0081] To enable the model to understand the described defect features, a standardized description template can be established. Table 1 illustrates a description template as an example. It should be understood that Table 1 is for reference only and does not constitute a limitation; in actual use, the description template should be designed based on the actual appearance and distribution of the defect. As shown in Table 1, the defect can be decomposed into the following dimensions for text encoding.

[0082] Table 1

[0083]

[0084] Taking a wafer as an example, the format of the defect description text is as follows:

[0085] “wafer surface with [shape] defect, appearing as [color] mark of [size] scale, located at [position], oriented [direction], with [edge features], showing [quantity] occurrence.”

[0086] A defect mask is a mask image that predefines the location, shape, and size of a defect. In a defect mask, foreground pixels have a grayscale value of 1, and background pixels have a grayscale value of 0. The defect mask is a single-channel image with the same dimensions as the original image. The defect mask is used to provide constraints on the shape, location, and size of defects when a defect image generation model generates defect images.

[0087] The diffusion model in the defect image generation model includes a feature extraction network and at least one decoder. The feature extraction network includes a variational autoencoder, at least one encoder, and an intermediate block.

[0088] like Figure 2 As shown in the dashed box 21, the framework of the defect image generation model is illustrated. The diffusion model includes a variational autoencoder (VAE Encoder), SD encoders A-D (i.e., four SD encoder blocks, namely SD Encoder Block A-SD Encoder Block D, where SD is an abbreviation for Stable Diffusion), an intermediate block (SDMiddle Block), SD decoders D-A (i.e., four SD decoder blocks, namely SD Decoder Block D-SD Decoder Block A), and a variational autodecoder (VAE Decoder). From the perspective of feature extraction, the variational autoencoder, SD encoders A-D, and the intermediate blocks belong to the feature extraction network 211. From the perspective of denoising processing of the diffusion model, SD encoders A-D, the intermediate blocks, and SD decoders D-A are the core backbone networks responsible for "denoising".

[0089] like Figure 2 As shown, the input to the diffusion model can also include a time step. During the model inference phase, the time step is controlled by user input and is used to characterize the noise concentration to be removed. The process parameters (i.e., target process parameters), time steps, and defect description text can be mapped into corresponding feature vectors through their respective encoders (i.e., parameter encoder, time encoder, and text encoder), and then used together with the input image (i.e., the base object graph) as input data for the diffusion model.

[0090] like Figure 2 As shown, the input image can first be compressed to a low-dimensional latent space by a variational autoencoder (for example, the size of the input image is 512x512, and the size of the compressed feature map is 64x64). The compressed feature map is then processed by the SD encoder A~D.

[0091] It should be understood that process parameters are transformed into parameter feature vectors by a parameter encoder. For example, the parameter encoder can reuse the Time Embedding MLP structure in the basic model framework (such as the ControlNet framework), that is, through two fully connected layers and the SiLU activation function, the process parameters are mapped to a high-dimensional space of a specified dimension (the dimension is the same as the number of channels of the image features to be fused), thus obtaining the parameter feature vector.

[0092] The defect description text is processed by a text encoder to generate a corresponding text feature vector.

[0093] The time step is converted into a time feature vector by a time encoder. Following the fusion method of the Time Encoder, both the parameter feature vector and the text feature vector can be fused into various modules of the core backbone network of the diffusion model based on the time feature vector. For example... Figure 2 As shown, the inputs are fed into four SD Encoder Blocks, four SD Middle Blocks, and four SD Decoder Blocks, respectively, to fuse with the input image features of each module, ultimately yielding the output features of each module. It should be noted that the number of channels in the input image features of each module may differ; therefore, the parameter feature vector and text feature vector input to each module have the same number of channels as the input image features of that module.

[0094] The conditional control network consists of a first zero-convolutional layer, four SD encoders A-D (i.e., copies of SD encoders A-D in the diffusion model), an intermediate block (i.e., a copy of the SD Middle Block in the diffusion model), and five second zero-convolutional layers. The output of each encoder and intermediate block in the conditional control network is connected to a second zero-convolutional layer, resulting in five second zero-convolutional layers. Figure 2 As shown, the defect mask image is input into the conditional control network and sequentially processed through the first zero convolutional layer, SD encoders A~D, intermediate blocks, and the second zero convolutional layer. Each second zero convolutional layer outputs the corresponding defect control feature.

[0095] It should be noted that the initial parameters of the SD encoders A~D and intermediate blocks in the conditional control network are the corresponding parameters of the SD encoders A~D and intermediate blocks in the diffusion model. Since the diffusion model is pre-trained, the parameters of the diffusion model will be kept locked when training the defect image generation model, and only the zero convolutional layer, SD encoders A~D and intermediate blocks in the conditional control network will be trained.

[0096] Please continue reading. Figure 2The features output by the last SD encoder (i.e., SD encoder D) in the feature extraction network 211 and the defect control features output by the second zero convolutional layer corresponding to the intermediate block of the conditional control network are input into the intermediate block in the diffusion model to obtain the target features output by the intermediate block.

[0097] In some embodiments, the target process parameter can be a process parameter input by the user. For example, the process parameter actually used in the current production line (i.e., the specific process parameter value) input by the user. It should be noted that the target process parameter and the predicted process parameter in the various embodiments of this application refer not only to process parameter items, but also to specific parameter values.

[0098] In some embodiments, the target process parameters are automatically predicted process parameters. That is, the target process parameters that guide the generation of defect images can be directly predicted without manual input by the user, greatly improving efficiency and saving manual operation costs. For example, the target process parameters are predicted by a process parameter prediction model in the target model based on a second target image. The second target image is a base object image or a defect reference image. The defect reference image is essentially a known defect image, meaning the target object in the defect reference image carries a defect. However, because the process parameters corresponding to the defect reference image can serve as target process parameters to guide the defect image generation model in generating defect images, it is called a defect reference image.

[0099] Figure 3 This is the model architecture of a process parameter prediction model in one embodiment. For example... Figure 3 As shown, the process parameter prediction model includes a feature extraction network 211 shared with the diffusion model and a parameter prediction module. Before executing step S12, the second target image (base object image or defect reference image) can be input into the feature extraction network 211 in the process parameter prediction model to obtain image features. That is, the intermediate block of the feature extraction network outputs image features. It should be understood that the input of the intermediate block of the process parameter prediction model is the output features of the last SD encoder, and does not involve the defect control features output by the conditional control network.

[0100] Then, the process parameters corresponding to the second target image are predicted based on the image features through the parameter prediction module; the process parameters predicted for the second target image are used as the target process parameters.

[0101] Please continue reading. Figure 3 The parameter prediction module converts the abstract visual feature vector (i.e., the image features mentioned above) output by the feature extraction network 211 into specific process parameters through two cascaded convolutional layers and a fully connected layer.

[0102] Specifically, taking a basic object image with an input image of 512x512x3 pixels as an example, the feature extraction network 211 performs feature extraction processing, and the intermediate block outputs a feature map (i.e., image features). First, the feature map undergoes feature extraction processing through the first convolutional layer: the feature map is processed by 640 convolutional kernels of size 1280x3x3 with a stride of 2, compressing the number of channels from 1280 to 640, while simultaneously reducing the size to 4x4. Then, it undergoes convolution processing through the second convolutional layer: 320 convolutional kernels of size 640x4x4 with a stride of 1, compressing the number of channels from 640 to 320, while simultaneously reducing the size to 1x1. Further, the feature map output from the second convolutional layer is flattened into a 320-dimensional vector, and finally mapped to an N-dimensional process parameter space through a fully connected layer, outputting the process parameters. Here, N corresponds to the preset process parameter dimension in the production process (i.e., the number of process parameter items such as exposure dose).

[0103] S14, through the decoder of the diffusion model, decodes based on target features and defect control features to generate a defect image of the target object.

[0104] Specifically, the target features output from the intermediate block and the defect control features output from the conditional control network can be input into the decoder in the diffusion model for decoding processing to generate a defect image of the target object. For example, there can be multiple decoders in the diffusion model.

[0105] Please continue reading. Figure 2 The target features output from the intermediate blocks of the diffusion model and the defect control features output from the second zero convolutional layers corresponding to each encoder in the conditional control network are input into the corresponding decoders in the diffusion model. For example, the target features output from the intermediate blocks of the diffusion model and the defect control features output from the second zero convolutional layers corresponding to the SD encoder D in the conditional control network are input into the corresponding SD decoder D in the diffusion model. For example, as... Figure 2 As shown, based on the time feature vector encoded at the time step, the parameter feature vector and text feature vector can also be input into each SD decoder. Each SD decoder performs decoding processing based on the multi-dimensional features of the input. After decoding by four decoders in sequence, the decoding result is input into the variational autodecoder to restore the image space before compression, and finally generate and output the defect image. That is, the defect image generation model outputs the defect image through a diffusion process under the guidance of process parameters. The output defect image not only conforms to the constraints of the preset defect mask in terms of geometric shape, but also more accurately reflects the physical characteristics under the process conditions in terms of texture, color difference and subtle features.

[0106] The above scheme achieves a leap from "visual fitting" to "physical simulation," enabling the generation of defect distributions to dynamically change with process conditions (such as temperature, pressure, and other process parameters), thus better reflecting physical realism. Furthermore, it allows for accurate and diverse control of defect image generation based on target process parameters. For example, when the user fine-tunes process parameters during the inference phase (e.g., increasing exposure dose), the generated defect image exhibits a corresponding visual shift (e.g., edges become sharper or blurrier). This parameter interpretability is completely lacking in traditional AI models that rely solely on mask control. In addition, some of the above embodiments can use a process parameter prediction model to predict the process parameters corresponding to the input image, realizing a bidirectional mapping logic between "process" and "image."

[0107] It should be understood that Figure 2 and Figure 3 The frameworks of the defect image generation model and the process parameter prediction model are illustrated separately. The target model can include both the defect image generation model and the process parameter prediction model; therefore, the overall framework of the target model can be found in [reference needed]. Figure 4 .from Figure 4 It is readily apparent that the target model is an improvement upon the ControlNet architecture. It reuses the basic encoder structure of the ControlNet architecture to form a parameter encoder, thereby enabling the encoding of process parameters. Furthermore, it reuses the feature extraction network 211 in the diffusion model of the defect image generation model, and extends the parameter prediction module after the middle block of the feature extraction network 211 to form a process parameter prediction model.

[0108] Using the Middle Block in the feature extraction network as an access point allows access to more informative and accurate feature maps, thereby improving the accuracy of process parameter prediction. Because the Middle Block is located at the bottom layer of the U-Net structure (i.e., the core backbone network), after sufficient downsampling of image features, its feature map already possesses highly compressed global semantic information, which can represent the overall environmental features of the defect generation environment.

[0109] In some embodiments, the generated defect images can be used as samples to train a defect detection model. Furthermore, the generated defect images can also be used in other scenarios such as simulations. For example, this method can be used for defect simulation applications in specific semiconductor processes such as photolithography, deposition, grinding, and cutting, covering specific defect types such as polishing debris, scratches, and bubbles.

[0110] For example, the target process parameters can be continuously modified or perturbed to attempt inputting process parameters at extreme edges, generating rare "edge cases"—that is, defect images generated under process parameters at extreme edges. For instance, when perturbing the target process parameters, the changes in defect morphology in the generated defect images can be observed, allowing for continuous perturbation and adjustment to approximate the target process parameters at extreme edges, thereby generating defect images under target process parameters at extreme edges. Because defect images under extremely abnormal process parameters (such as extremely high pressure) in real production are extremely difficult to obtain, the method of this application can obtain defect images under extreme process parameters.

[0111] It should be understood that generating defect images under extreme process parameters plays a very important role and has great significance. Taking the defect detection scenario as an example, using these defect images as samples to train the defect detection model can significantly improve the defect detection model's ability to alarm against sudden and extreme process runaway, realizing the transformation from "passive identification" to "active defense".

[0112] In some embodiments, unsupervised state monitoring capabilities can also be achieved through process parameter prediction models to monitor the state of equipment (i.e., computer equipment that performs the methods in the various embodiments of this application).

[0113] Specifically, in this embodiment, the target process parameter is the input process parameter. During the model's inference phase, the process parameter prediction model can predict the process parameters corresponding to the first target image; further, the predicted process parameters for the first target image can be compared with the input process parameters. If the difference between the predicted process parameters for the first target image and the target process parameter exceeds a preset deviation range, a prompt message is generated. For example, this prompt message is used to indicate a potential deviation in the equipment state. The output method of the prompt message is not limited in this embodiment.

[0114] The first target image includes a base object graph or a defect image generated from the model.

[0115] In some examples, the first target image is the base object map, and the target process parameters are the actual process parameters corresponding to the input base object map. In this case, since the process parameter prediction model and the defect image generation model share the feature extraction network, the feature extraction process does not need to be repeated when predicting process parameters. Instead, the image features extracted from the base object map by the feature extraction network of the diffusion model during defect image generation are directly used and input into the parameter prediction module of the process parameter prediction model to predict the corresponding process parameters. This saves computational resources and improves efficiency. It should be noted that the image features input to the parameter prediction module are only the image features obtained by the diffusion model from the feature extraction of the base object map, and do not involve the defect control features output by the conditional control network.

[0116] In some examples, the target process parameter is a specified input process parameter, and the first target image is a defect image generated by the defect image generation model in step S14. In this example, the defect image generation model in the target model is not initially disabled and can be used to generate defect images. After the defect image is generated, the defect image generation model is disabled, and the process parameter prediction model can still work independently. Furthermore, the defect image generated in step S14 can be input into the feature extraction network in the process parameter prediction model for feature extraction. The extracted image features are then input into the parameter prediction module in the process parameter prediction model to predict the corresponding process parameters.

[0117] In other examples, the first target image can also be any specified image with defects or without defects. Specifically, the defect image generation model in the target model can always be disabled, and the process parameter prediction model can work independently to monitor the equipment status, thereby predicting the corresponding process parameters for any input first target image and performing subsequent equipment status alerts.

[0118] In some embodiments, the ability to locate process parameter offsets can also be achieved based on a process parameter prediction model. Specifically, due to factors such as equipment aging, cavity contamination, and sensor offset, the process parameter values ​​used in actual production often differ from the set values ​​(i.e., the input process parameter values). When equipment frequently experiences abnormal defects, the process parameter prediction model can be used to predict process parameters (i.e., estimated values) based on input images collected in the current production environment. This allows engineers to troubleshoot based on the direction and magnitude of the offset between the estimated and set values ​​of the process parameters, enabling faster problem location and reducing the time cost of problem localization.

[0119] The training process of the target model will be described below. Specifically, the target model can be trained in two ways: phased training (referred to as Method 1) or end-to-end joint training (referred to as Method 2).

[0120] Method 1: Phased training.

[0121] In Method 1, the training phases of the process parameter prediction model and the defect image generation model in the target model are separate. Specifically, the process parameter prediction model in the target model is first trained using a small number of defect sample images carrying process parameter labels and free of noise. Since data gaps can easily occur, potentially resulting in a large number of defect sample images lacking process parameters, the trained process parameter prediction model can be used to automatically and accurately predict and complete the missing process parameter labels for defect sample images, ensuring they match the product's appearance. Based on these predicted process parameters, the defect sample images are effectively expanded, improving expansion efficiency and reducing sample costs. Furthermore, the large number of expanded and completed defect sample images carrying process parameter labels can then be used to train the defect image generation model, thereby effectively improving its accuracy.

[0122] In the embodiments of this application, the defect sample image refers to a surface image of a target object with defects, used for model training. For example, an imaging image of a defective wafer surface can be used as a defect sample image for model training.

[0123] Specifically, the training process of the target model includes a first training phase for training the process parameter prediction model and a second training phase for training the defect image generation model.

[0124] In the first training phase, the process parameter prediction model is trained iteratively using first defect sample images that carry process parameter labels but do not carry noise. For example, in each training round, the process parameter prediction model is used to predict the corresponding process parameters for each first defect sample image, the parameter loss is calculated based on the difference between the predicted process parameters and the corresponding process parameter labels, and the parameters of the parameter prediction module in the process parameter prediction model are adjusted based on the parameter loss.

[0125] It should be noted that since the process parameter prediction model shares the feature extraction network of the diffusion model, the parameters (i.e., weight parameters) of the feature extraction network need to be frozen or locked during the training of the process parameter prediction model. Instead, the parameters of the parameter prediction module in the process parameter prediction model are trained separately. After the first stage of training is completed, the process parameter prediction model has the ability to map image features to process parameter values ​​and can accurately predict the corresponding process parameters for the input image.

[0126] Between the first and second training phases, a sample augmentation and completion phase is included. In this phase, a second defect sample image without missing process parameters and carrying no noise is acquired. This second defect sample image has corresponding defect text annotations (i.e., the annotated defect description text) and defect pixel-level annotations (i.e., the annotated defect mask image), but lacks process parameters. Therefore, the trained process parameter prediction model is used to predict the corresponding first process parameters for the input second defect sample image without missing process parameters and carrying no noise, and the first process parameters are used as the process parameter labels for the second defect sample image. Based on the first and second defect sample images, a first sample set carrying noise is formed, for example, by adding noise (such as Gaussian noise) to the first and second defect sample images. Each defect sample image in the first sample set carries noise. This achieves accurate and effective sample augmentation, effectively increasing the sample size for the subsequent second training phase.

[0127] It should be understood that the diffusion model in the defect image generation model to be trained is a pre-trained model that already possesses image generation capabilities. To enable the diffusion model to understand physical processes without compromising its image generation capabilities, a weight parameter locking training strategy was adopted in the second training phase. That is, the pre-trained diffusion model was locked and no longer trained (e.g., locked...). Figure 2 In the diffusion model, all pre-trained weight parameters of the encoders, decoders, intermediate blocks, etc., are not adjusted during training. Instead, the defect sample images from the first sample set and their corresponding first defect annotation information are used to iteratively train the conditional control network in the defect image generation model to be trained (i.e., adjusting the parameters of the first / second zero convolutional layers, encoders, intermediate blocks, etc. in the conditional control network), resulting in the trained defect image generation model. The first defect annotation information includes process parameter labels corresponding to the defect sample images, defect text annotations, and defect pixel-level annotations. These three types of annotations are equivalent to three different dimensions of defect generation constraints.

[0128] Similar to the principles of the model inference stage, it can also be combined with Figure 2 To understand this, during the second training phase of training the defect image generation model, process parameter labels, time steps, defect text annotations (i.e., labeled defect description text), and input images (i.e., defect sample images) are input into the diffusion model to be trained; defect pixel-level annotations (i.e., labeled defect mask images) are input into the conditional control network to be trained. The time step in the second training phase is a randomly generated integer by the model, representing the noise concentration applied by the model to the original image (i.e., the defect sample image).

[0129] Specifically, in the second training phase, in each round of training of the defect image generation model, based on the defect image generation model to be trained in this round, defect images corresponding to the defect sample images in the first sample set are generated. The principle of generating defect images in the second training phase is the same as in the model inference phase, and the specific details of defect image generation will not be elaborated here. Unlike the inference phase, in the second training phase, it is necessary to calculate the noise loss based on the difference between the noise predicted when generating the defect image and the noise carried by the defect sample images in the first sample set, and adjust the parameters of the conditional control network in the defect image generation model according to the noise loss.

[0130] Method 2: End-to-end joint training.

[0131] In Method 2, the process parameter prediction model and the defect image generation model in the target model do not need to be trained separately, but are trained jointly.

[0132] Specifically, a second sample set is obtained, which contains a third defect sample image carrying second defect annotation information and noise; the second defect annotation information includes process parameter labels, defect text annotations, and defect pixel-level annotations corresponding to the third defect sample image.

[0133] In each round of model training, based on the defect image generation model to be trained in this round, a first defect image corresponding to each third defect sample image is generated; based on the process parameter prediction model to be trained in this round, the second process parameters corresponding to each third defect sample image are predicted; based on the noise loss and parameter loss corresponding to each third defect sample image, the corresponding target loss is obtained; the noise loss represents the difference between the noise predicted when generating the first defect image and the noise carried by the third defect sample image, and the parameter loss represents the difference between the second process parameter corresponding to the third defect sample image and the process parameter label; based on the target loss corresponding to each third defect sample image, the total loss is calculated; based on the total loss, the parameters of the conditional control network in the process parameter prediction model and the defect image generation model to be trained in this round are adjusted, and the next round of model training begins, until the target model is obtained after training is completed.

[0134] In some embodiments, in each round of model training, the second process parameter is used as a new process parameter label, and the defect image generation model is used to regenerate the image based on the updated process parameter label to generate the second defect image corresponding to each third defect sample image; the reconstruction feature loss is obtained based on the feature difference between the second defect image and the first defect image corresponding to each third defect sample image; and then, the target loss corresponding to each third defect sample image is obtained based on the noise loss, parameter loss and reconstruction feature loss corresponding to each third defect sample image.

[0135] In some embodiments, the second sample set further includes a fourth defect sample image carrying third defect annotation information and noise; the second defect annotation information includes defect text annotation and defect pixel-level annotation, but does not include process parameter labels.

[0136] In other words, the second sample set includes both the third defect sample images carrying complete defect labeling information and noise (which can be denoted as: the second type of data with process parameter labels) and the fourth defect sample images that lack process parameter labels but have other defect labeling information and noise (which can be denoted as: the first type of data without process parameter labels).

[0137] Thus, in each round of model training, for the fourth defect sample image (i.e., the first type of data without process parameter labels), corresponding first defect images, second defect images, and second process parameters are also generated. Specifically, in the case of missing process parameter labels, the first defect image corresponding to each fourth defect sample image is first generated based on the defect image generation model to be trained in this round; then, based on the process parameter prediction model to be trained in this round, the second process parameters corresponding to each fourth defect sample image are predicted, and the predicted second process parameters are used as the process parameter labels corresponding to the fourth defect sample images. The defect image generation model is then used to generate the second defect images corresponding to each fourth defect sample image based on the corresponding process parameter labels.

[0138] Furthermore, in each round of model training, based on the process parameter prediction model to be trained in this round, the third process parameter can be predicted for the second defect image corresponding to the fourth defect sample image; the prediction stability representation loss is determined according to the difference between the third process parameter and the second process parameter corresponding to each fourth defect sample image. Then, based on the noise loss and prediction stability representation loss corresponding to each fourth defect sample image, the target loss corresponding to each fourth defect sample image is obtained.

[0139] Finally, the target loss corresponding to "Type 2 data with process parameter labels" can be combined with the target loss corresponding to "Type 1 data without process parameter labels" to calculate the total loss. That is, the total loss is calculated based on the target losses corresponding to each third defect sample image and each fourth defect sample image.

[0140] In some embodiments, the loss functions for the first type of data without process parameter labels and the second type of data with process parameter labels are as follows:

[0141] ;

[0142]

[0143]

[0144]

[0145] in, This is either the noise loss function or the generation loss function; The parameter loss function characterizes the process parameter labels. With the predicted second process parameters The differences between them; To reconstruct the feature loss function, used to characterize "labeled with real process parameters". The first defect image generated "and "using the predicted second process parameters The generated second defect image "Feature differences in high-dimensional feature space," This represents the third defect sample image input. This represents the feature map function of each SD Decoder Block D~A, therefore... Image representing the first defect Features Image representing the second defect The characteristics of "". The loss function is used to characterize the predictive stability of the process parameter prediction model. Indicates the second defect image The predicted third process parameters; Indicates the predicted second process parameter With the third process parameters The differences between them.

[0146] It should be understood that, through It can constrain "using actual process parameter labels" The first defect image generated "and "using the predicted second process parameters The generated second defect image "Maintaining consistency or near consistency in the high-dimensional feature space can indirectly constrain the prediction of the second process parameter." Approximate process parameter labels The effect.

[0147] Furthermore, the second process parameters will be based on predictions. The generated second defect image The second parameter prediction is performed again through the parameter prediction branch (i.e., the process parameter prediction model) to obtain the third process parameter predicted in the second prediction. ,pass Constraints on third process parameters Compared with the first predicted second process parameters Maintaining consistency or near consistency is equivalent to adding self-supervised constraints to the first type of data without process parameter labels, which can significantly improve the stability and rationality of the prediction branch in the case of no true value (i.e. no process parameter labels).

[0148] In the above scheme, the second type of data with process parameter labels directly supervises parameter prediction and provides a consistency constraint signal between "real conditions" and "predicted conditions" for image generation processing. The first type of data without process parameter labels is processed through noise loss. (Also known as generation loss) and prediction stability characterization loss The training signal is fed back into the parameter prediction branch, enabling the process parameter prediction model to "learn" to output process parameter values ​​useful for the generation task. This process parameter prediction model establishes a nonlinear mapping relationship between low-level image features and physical process variables. Based on this nonlinear mapping, the model can map input image features to the feature space. In short, the process parameter prediction model is implicitly trained while serving the generation task, and the defect image generation model receives better input due to more reasonable predicted process parameters. The accuracy of process parameter prediction and the quality of defect generation mutually promote each other, co-evolving to achieve better end-to-end training. This improves the accuracy of both the defect image generation model and the process parameter prediction model, leading to more accurate defect image generation. Furthermore, end-to-end training can improve model training efficiency to some extent.

[0149] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. At least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the steps or stages of other steps.

[0150] Based on the same inventive concept, this application also provides an apparatus for generating defect images as described above. The solution provided by this apparatus is similar to the solution described in the above method; therefore, the specific limitations of one or more defect image generation apparatus embodiments provided below can be found in the limitations of the defect image generation method described above, and will not be repeated here.

[0151] In one embodiment, such as Figure 5 As shown, a defect image generation apparatus is provided, the apparatus comprising:

[0152] The model determination unit 502 is used to load the trained target model; the target model includes a defect image generation model; the defect image generation model includes a conditional control network and a diffusion model.

[0153] The feature extraction unit 504 is used to input the target process parameters, the basic object diagram of the target object without defects, and the preset defect description text into the feature extraction network of the diffusion model for feature encoding to obtain the target features.

[0154] The condition control unit 506 is used to input a preset defect mask image into the condition control network to obtain defect control features.

[0155] Image reconstruction unit 508 is used to generate a defect image of the target object by decoding based on target features and defect control features through a decoder of a diffusion model.

[0156] In some embodiments, the target model further includes a process parameter prediction model; the target process parameters are the input process parameters; the apparatus further includes:

[0157] The process parameter prediction unit 503 is used to predict the process parameters corresponding to the first target image through the process parameter prediction model; the first target image includes a base object image or a defect image.

[0158] The reminder unit 510 is used to generate a reminder message when the difference between the process parameters predicted for the first target image and the target process parameters exceeds a preset deviation range.

[0159] In some embodiments, the training process of the target model includes a first training phase for training a process parameter prediction model and a second training phase for training a defect image generation model. For example... Figure 6 As shown, the device also includes: a training unit 501, a process parameter prediction unit 503, and a reminder unit 510; wherein:

[0160] Training unit 501 is used to train a process parameter prediction model using a first defect sample image carrying process parameter labels but without noise in the first training phase; using the trained process parameter prediction model to predict the corresponding first process parameter for a second defect sample image with missing process parameters but without noise, and using the first process parameter as the process parameter label of the second defect sample image; forming a first sample set carrying noise based on the first and second defect sample images; in the second training phase, locking the pre-trained diffusion model and not training it, and using the defect sample images in the first sample set and the corresponding first defect annotation information to train the conditional control network to obtain a trained defect image generation model; wherein, the first defect annotation information includes the process parameter label corresponding to the defect sample image, the defect text annotation, and the defect pixel-level annotation.

[0161] In some embodiments, the training unit 501 is further configured to acquire a second sample set containing third defect sample images carrying second defect annotation information and noise; the second defect annotation information includes process parameter labels, defect text annotations, and defect pixel-level annotations corresponding to the third defect sample images; in each round of model training, based on the defect image generation model to be trained in this round, a first defect image corresponding to each third defect sample image is generated; based on the process parameter prediction model to be trained in this round, the second process parameters corresponding to each third defect sample image are predicted; based on the noise loss and parameter loss corresponding to each third defect sample image, the corresponding target loss is obtained; the noise loss characterizes the difference between the noise predicted when generating the first defect image and the noise carried by the third defect sample image, and the parameter loss characterizes the difference between the second process parameters corresponding to the third defect sample image and the process parameter labels; based on the target loss corresponding to each third defect sample image, the total loss is calculated; based on the total loss, the parameters of the conditional control network in the process parameter prediction model and the defect image generation model to be trained in this round are adjusted, and the next round of model training is entered until the target model is obtained after training is completed.

[0162] In some embodiments, the training unit 501 is further configured to, in each round of model training, use the second process parameter as a new process parameter label, use the defect image generation model, regenerate the second defect image corresponding to each third defect sample image based on the updated process parameter label; obtain the reconstruction feature loss based on the feature difference between the second defect image and the first defect image corresponding to each third defect sample image; and obtain the corresponding target loss based on the noise loss, parameter loss and reconstruction feature loss corresponding to each third defect sample image.

[0163] In some embodiments, the second sample set further includes a fourth defect sample image carrying third defect annotation information and noise; the third defect annotation information includes defect text annotation and defect pixel-level annotation, but does not include process parameter labels; in each round of model training, a corresponding first defect image, a second defect image, and a second process parameter are also generated for the fourth defect sample image; the training unit 501 is further configured to predict the third process parameter for the second defect image corresponding to the fourth defect sample image in each round of model training, based on the process parameter prediction model to be trained in this round; determine the prediction stability representation loss according to the difference between the third process parameter and the second process parameter corresponding to each fourth defect sample image; obtain the corresponding target loss according to the noise loss and prediction stability representation loss corresponding to each fourth defect sample image; and calculate the total loss according to the target losses corresponding to each third defect sample image and each fourth defect sample image respectively.

[0164] In some embodiments, the target model further includes a process parameter prediction model; the process parameter prediction model includes a feature extraction network shared with the diffusion model, and a parameter prediction module;

[0165] The process parameter prediction unit 503 is also used to input the second target image into the feature extraction network in the process parameter prediction model to obtain image features; the second target image includes a base object image or a defect reference image; the parameter prediction module predicts the process parameters corresponding to the second target image based on the image features; and the process parameters predicted for the second target image are used as the target process parameters.

[0166] Each unit in the aforementioned defect image generation device can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each unit.

[0167] In one embodiment, a computer device is provided, which is a testing machine or other electronic device with defect image generation capabilities. The internal structure diagram of the computer device can be as follows: Figure 7As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external devices via a network connection. When the computer program is executed by the processor, it implements a defect image generation method.

[0168] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0169] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the embodiments of this application.

[0170] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the embodiments of this application.

[0171] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. The processors involved in the embodiments provided in this application can be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited thereto.

[0172] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for generating defect images, characterized in that, The method includes: Load the trained target model; the target model includes a defect image generation model; the defect image generation model includes a conditional control network and a diffusion model; The target process parameters, the basic object diagram of the target object without defects, and the preset defect description text are input into the feature extraction network of the diffusion model for feature encoding to obtain the target features; The preset defect mask image is input into the conditional control network to obtain defect control features; The diffusion model decoder performs decoding processing based on the target features and the defect control features to generate a defect image of the target object.

2. The method according to claim 1, characterized in that, The target model further includes a process parameter prediction model; the target process parameters are the input process parameters; the method further includes: The process parameters corresponding to the first target image are predicted using the process parameter prediction model; the first target image includes the base object image or the defect image. If the difference between the process parameters predicted for the first target image and the target process parameters exceeds a preset deviation range, a prompt message is generated.

3. The method according to claim 2, characterized in that, The training process of the target model includes a first training phase for training the process parameter prediction model and a second training phase for training the defect image generation model. The training steps for the target model include: In the first training phase, the process parameter prediction model is trained only using a first defect sample image that carries process parameter labels but does not carry noise. The trained process parameter prediction model is used to predict the corresponding first process parameter for the second defect sample image with missing process parameters and no noise. The first process parameter is used as the process parameter label of the second defect sample image. A first sample set with noise is formed based on the first defect sample image and the second defect sample image. In the second training phase, the pre-trained diffusion model is locked and not trained, and the conditional control network is trained using the defect sample images in the first sample set and the corresponding first defect annotation information to obtain the trained defect image generation model; wherein, the first defect annotation information includes the process parameter labels, defect text annotations and defect pixel-level annotations corresponding to the defect sample images.

4. The method according to claim 2, characterized in that, The training steps for the target model include: Obtain a second sample set containing a third defect sample image carrying second defect annotation information and noise; the second defect annotation information includes process parameter labels, defect text annotations, and defect pixel-level annotations corresponding to the third defect sample image; In each round of model training, based on the defect image generation model to be trained in this round, a first defect image corresponding to each of the third defect sample images is generated; Based on the process parameter prediction model to be trained in this round, predict the second process parameters corresponding to each of the third defect sample images; Based on the noise loss and parameter loss corresponding to each third defect sample image, the corresponding target loss is obtained; the noise loss represents the difference between the noise predicted when generating the first defect image and the noise carried by the third defect sample image, and the parameter loss represents the difference between the second process parameter and the process parameter label corresponding to the third defect sample image. Calculate the total loss based on the target loss corresponding to each of the aforementioned third defect sample images; Based on the total loss, the conditional control network in the process parameter prediction model and defect image generation model to be trained in this round is adjusted, and the next round of model training begins until the target model is obtained after training is completed.

5. The method according to claim 4, characterized in that, In each round of model training, the method further includes: Using the second process parameter as a new process parameter label, the defect image generation model is used to regenerate the second defect image corresponding to each of the third defect sample images based on the updated process parameter label. Based on the feature differences between the second defect image and the first defect image corresponding to each of the third defect sample images, the reconstruction feature loss is obtained; The target loss obtained based on the noise loss and parameter loss corresponding to each third defect sample image includes: Based on the noise loss, parameter loss, and reconstruction feature loss corresponding to each third defect sample image, the corresponding target loss is obtained.

6. The method according to claim 5, characterized in that, The second sample set also includes a fourth defect sample image carrying third defect annotation information and noise; the third defect annotation information includes defect text annotation and defect pixel-level annotation, but does not include process parameter labels; in each round of model training, the corresponding first defect image, second defect image, and second process parameter are also generated for the fourth defect sample image; The method further includes: In each round of model training, based on the process parameter prediction model to be trained in this round, the third process parameter is predicted for the second defect image corresponding to the fourth defect sample image. The predicted stability characterization loss is determined based on the difference between the third process parameter and the second process parameter corresponding to each fourth defect sample image. Based on the noise loss corresponding to each of the fourth defect sample images and the predicted stability characterization loss, the corresponding target loss is obtained; The calculation of the total loss based on the target loss corresponding to each of the third defect sample maps includes: The total loss is calculated based on the target loss corresponding to each of the third defect sample maps and each of the fourth defect sample maps.

7. The method according to claim 1, characterized in that, The target model further includes a process parameter prediction model; the process parameter prediction model includes the feature extraction network and parameter prediction module shared with the diffusion model; Before inputting the target process parameters, the base object diagram of the target object without defects, and the preset defect description text into the feature extraction network of the diffusion model for feature encoding to obtain the target features, the method further includes: The second target image is input into the feature extraction network in the process parameter prediction model to obtain image features; the second target image includes the base object image or the defect reference image. The parameter prediction module predicts the process parameters corresponding to the second target image based on the image features. The process parameters predicted for the second target image are used as the target process parameters.

8. A defect image generation apparatus, characterized in that, The device includes: A model determination unit is used to load the trained target model; the target model includes a defect image generation model; the defect image generation model includes a conditional control network and a diffusion model. The feature extraction unit is used to input the target process parameters, the basic object diagram of the target object without defects, and the preset defect description text into the feature extraction network of the diffusion model for feature encoding to obtain the target features; The condition control unit is used to input a preset defect mask image into the condition control network to obtain defect control features; The image reconstruction unit is used to generate a defect image of the target object by performing decoding processing based on the target features and the defect control features through the decoder of the diffusion model.

9. A computer device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images