Image generation method and device, computer device and computer readable storage medium

Abstract

The application relates to an image generation method and device, computer equipment and a computer readable storage medium. The method comprises the following steps: acquiring sample defect images of the same object under multiple candidate visual angles respectively; training a preset image generation model by taking a sample non-defect image of the same object under a first visual angle as input and a sample defect image under the first visual angle as output, to obtain a first defect image generation model corresponding to the first visual angle; the first defect image generation model is used for generating a defect image under the first visual angle according to the non-defect image under the first visual angle; and the image generation model is trained by taking the sample defect image of the same object under the first visual angle as input and a sample defect image under a second visual angle as output, to obtain a second defect image generation model corresponding to the second visual angle. According to the application, the accuracy of multi-angle defect image generation can be improved.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to an image generation method, apparatus, computer device, and computer-readable storage medium. Background Technology

[0002] In the training of defect detection models, due to the lack or difficulty in obtaining real defect samples, sample images containing defects can be generated to enrich the sample size. Furthermore, for the purpose of more accurate defect detection, defect images of the same object from multiple different viewpoints can be generated.

[0003] In related technologies, the input layer of the generative model is usually modified directly to adapt to multi-channel stitched images. However, this method suffers from problems such as inconsistent positions and feature mismatches in the generated images across multiple viewpoints, resulting in poor simulation effects.

[0004] Therefore, a more accurate multi-view defect image generation scheme is needed. Summary of the Invention

[0005] Therefore, it is necessary to provide an image generation method, apparatus, computer device, computer-readable storage medium, and computer program product to address the aforementioned technical problems, thereby improving the accuracy of generating multi-view defect images.

[0006] In a first aspect, this application provides an image generation method, including:

[0007] Obtain sample defect images of the same object from multiple candidate viewpoints;

[0008] The image generation model is trained by taking the sample undefected images of the same object from a first perspective as input and the sample defective images from the first perspective as output, to obtain the first defective image generation model corresponding to the first perspective; the first defective image generation model is used to generate defective images from the first perspective based on the undefected images from the first perspective; the first perspective is any one of multiple candidate perspectives.

[0009] The image generation model is trained by taking the sample defect images of the same object from the first perspective as input and the sample defect images from the second perspective as output, to obtain the second defect image generation model corresponding to the second perspective; the second defect image generation model is used to generate defect images from the second perspective based on the defect-free images from the first perspective; the second perspective is any one of the multiple candidate perspectives other than the first perspective.

[0010] Secondly, this application provides an image generation apparatus, comprising:

[0011] The acquisition module is used to acquire sample defect images of the same object from multiple candidate viewpoints.

[0012] The first training module is used to train a preset image generation model with sample undefected images of the same object from a first perspective as input and sample defective images from the first perspective as output, to obtain a first defective image generation model corresponding to the first perspective; the first defective image generation model is used to generate defective images from the first perspective based on the undefected images from the first perspective; the first perspective is any one of multiple candidate perspectives.

[0013] The second training module is used to train the image generation model with sample defect images of the same object from the first perspective as input and sample defect images from the second perspective as output, so as to obtain the second defect image generation model corresponding to the second perspective; the second defect image generation model is used to generate defect images from the second perspective based on the defect-free images from the first perspective; the second perspective is any one of the multiple candidate perspectives other than the first perspective.

[0014] Thirdly, this application provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the method described above.

[0015] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method.

[0016] Fifthly, this application provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the method described above.

[0017] The aforementioned image generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product train an image generation model in two steps. First, a first-view defect generation model is trained, taking a defect-free image from the first viewpoint as input and outputting a defect image from the same viewpoint. Then, a second-view defect generation model is trained, taking a defect image from the first viewpoint as input and outputting a defect image from the second viewpoint. This enables the generation of multi-view defect images from a defect-free image from the first viewpoint, ensuring consistency and continuity of defects across multiple views. Unlike related technologies where the location and shape of defects generated from the same material at different views deviate, failing to guarantee spatial consistency across multiple views, this embodiment selects a single viewpoint (such as the first viewpoint) from multiple candidate viewpoints as an anchor point, constructing a cascaded generation model from the anchor point viewpoint to other viewpoints (such as various second viewpoints). This effectively ensures the spatiotemporal consistency of defects across multiple views, providing support for more accurate defect detection. Attached Figure Description

[0018] Figure 1 This application provides an illustration of an image generation method.

[0019] Figure 2 A flowchart illustrating an image generation method provided in an embodiment of this application;

[0020] Figure 3 This is a structural block diagram of an image generation apparatus provided in an embodiment of this application;

[0021] Figure 4 An internal structural diagram of a computer device provided in an embodiment of this application;

[0022] Figure 5 An internal structural diagram of a computer device provided in another embodiment of this application;

[0023] Figure 6 This is an internal structural diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] The image generation method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a communication network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0026] like Figure 2 As shown, this application provides an image generation method, which is applied to... Figure 1 The method will be illustrated using terminal 102 or server 104 as examples. It is understood that the computer device may include at least one of a terminal and a server. The method includes the following steps:

[0027] S202: Obtain sample defect images of the same object from multiple candidate viewpoints.

[0028] The same object can include a unique physical item in the physical world. For example, in semiconductor inspection, the object can be a specific wafer; in bottled goods inspection, the object can be a specific glass bottle.

[0029] Candidate viewpoints refer to different ways of observing the same object from different physical angles or imaging conditions. For example, different viewpoints can be used to photograph a mobile phone glass cover using a camera directly above (candidate viewpoint A), a camera tilted 30 degrees to the left (candidate viewpoint B), and a camera tilted 30 degrees to the right (candidate viewpoint C) to capture scratches on the front and chipped edges.

[0030] Candidate viewpoints under different imaging conditions can be obtained by switching between bright field illumination and dark field illumination at the same camera position. The former can highlight surface contamination of the object, while the latter can highlight tiny scratches on the object. These two lighting conditions constitute two different candidate viewpoints.

[0031] Sample defect images refer to real-world defect images that have been collected and labeled into the training dataset. For example, for a scratched glass cover, the images showing the scratch taken by three cameras (A, B, and C, with each camera corresponding to a candidate viewpoint) constitute three sample defect images.

[0032] By acquiring sample defect images of the same object from multiple candidate perspectives, standard real training samples are provided for the subsequent training process of the defect sample generation model, thereby ensuring that the model learns how to generate images of real and accurate defect morphology.

[0033] S204: Train the preset image generation model with the sample undefected images of the same object under the first viewpoint as input and the sample defective images under the first viewpoint as output to obtain the first defective image generation model corresponding to the first viewpoint; the first defective image generation model is used to generate the defective image under the first viewpoint based on the undefected image under the first viewpoint; the first viewpoint is any one of multiple candidate viewpoints.

[0034] The first-person perspective serves as the starting point and benchmark for generating subsequent cascaded defect images. For example, candidate perspective A (directly above vertically) can be selected as the first-person perspective.

[0035] The input to the first defect image generation model corresponding to the first viewpoint is an image of the same object (such as a scratched glass cover) taken from viewpoint A when it is intact, i.e., a sample defect-free image (denoted as OK image). The output target of the first defect image generation model corresponding to the first viewpoint is a sample defect image of the same object from viewpoint A (denoted as NG image). The training process of the first defect image generation model corresponding to the first viewpoint can be to use multiple data pairs consisting of two types of images, such as viewpoint A_OK image and viewpoint A_NG image, to train a basic image generation model (e.g., a diffusion model not trained on a specific viewpoint), thereby obtaining the first defect image generation model corresponding to the first viewpoint. The learning target of this model can be, given an OK image from this viewpoint (i.e., the first viewpoint), to predict and generate a defect image that is consistent with the actual defect location and shape. After training, the first viewpoint defect image generation model (model G_A) is obtained. After training, model G_A has the ability to independently generate first-view defects. For example, when a new, unknown glass cover plate is input as a defect-free image under view A, model G_A can synthesize a defect image that meets the ideal simulation effect under that view. Thus, a model that can generate high-quality defect images based on defect-free images within this view (anchor point view) is obtained.

[0036] S206: The image generation model is trained with sample defect images of the same object from the first perspective as input and sample defect images from the second perspective as output, to obtain the second defect image generation model corresponding to the second perspective; the second defect image generation model is used to generate defect images from the second perspective based on the defect-free images from the first perspective; the second perspective is any one of the multiple candidate perspectives other than the first perspective.

[0037] In order to establish a defect information transmission channel from the anchor point perspective to other perspectives, thereby ensuring the spatiotemporal consistency of defect features of the same object in defect images under multiple perspectives, in this embodiment of the application, the image generation model is trained with the sample defect image of the same object under the first perspective as input and the sample defect image of the same object under the second perspective as output, so that the image corresponding to the second perspective can generate a consistent defect image under its own perspective based on the defect performance of the anchor point perspective (such as the first perspective) selected in the aforementioned steps.

[0038] The input to the second defect image generation model corresponding to the second perspective can include sample defect images of the same object obtained in the previous steps under the first perspective (view A), and its output target can be sample defect images of the same object under the second perspective (view B). Specifically, the training process of the second defect image generation model corresponding to the second perspective can include: using (view A_NG image, view B_NG image) data pairs to train a new image generation model (which can have the same structure as the aforementioned G_A, but with independent parameters). It can be understood that, unlike the training process of the first defect image generation model corresponding to the first perspective, the defect image generation model of the second perspective no longer learns defect generation "from nothing to something", but rather the perspective transformation and alignment "from the defect morphology of view A to the defect morphology of view B". After training, the second perspective defect image generation model (denoted as model G_B) is obtained. It should be noted that the direct input of model G_B is the defect image of view A. However, at the system level, to generate the defect of view B, the starting point is indeed the defect-free image of view A, and its workflow can be as follows:

[0039] First, model G_A generates a defective image from viewpoint A based on the defect-free image from viewpoint A. Then, the defective image from viewpoint A output by model G_A is used as input to model G_B. Finally, model G_B outputs the defective image from viewpoint B, thus realizing the function of multi-view defect image generation, which generates a defective image from a second viewpoint based on the defect-free image from the first viewpoint.

[0040] The cascaded training strategy adopted in this application embodiment "anchors" the generation behavior of model G_B (and similarly model G_C, etc.) to the output of model G_A. This ensures that when the model is applied, as long as model G_A generates a defect at a certain position, model G_B will generate a morphologically matching defect at the corresponding position under view B based on the learned cross-view mapping relationship. This ensures the consistency of defect generation under multiple views and solves the problem of defect position misalignment and morphological mismatch caused by multiple models generating independently.

[0041] In some embodiments, the image generation model includes a diffusion model; the diffusion model includes an encoder, an attention module, and a decoder connected in sequence.

[0042] The encoder is used to downsample the input image to obtain the image features of the input image; the attention module is used to fuse the defect annotation information, viewpoint feature information and image features of the input image based on the cross-attention mechanism to obtain the processed feature map.

[0043] The decoder is used to upsample the processed feature map to obtain the defect image corresponding to the input image; wherein, the defect annotation information includes at least one of the defect location annotation and defect type annotation of the input image.

[0044] The image generation model can be a diffusion model based on the Denoising Diffusion Probabilistic Model (DDPM).

[0045] The encoder can be the downsampling part of a U-Net-type network structure. It can consist of multiple cascaded convolutional and pooling layers (or using strided convolutions). The encoder's operation can include: assuming an input 512x512 pixel RGB image, the encoder progressively downsamples it to lower resolution, higher channel count feature maps such as 256x256, 128x128, and 64x64. The encoder captures multi-scale features of the input image, from local details to global semantics. The encoder's output can be a series of low-resolution, high-dimensional feature maps rich in semantic information.

[0046] The attention module receives image features from the encoder and incorporates external conditional information for guidance. Image features are feature maps output from the encoder. External conditional information may include defect annotations, which provide prior knowledge such as defect location and type to improve the simulation effect of the generated defect images. Defect location annotations can be represented as binary masks, where defect areas are represented by 1 (white) and non-defect areas by 0 (black), guiding the model to generate defects only in specified areas and avoiding erroneous modifications to normal areas. Defect type annotations can be represented as text labels (e.g., "scratches," "dents") or embedded vectors, guiding the model to generate specific types of defect morphologies.

[0047] Viewpoint feature information can be a feature vector representing the viewpoint to which the current image belongs, used to guide the model to generate defects that conform to the observation characteristics of that viewpoint. For example, for a tilted viewpoint, the generated defects should have corresponding perspective distortion and lighting effects.

[0048] The feature fusion process based on the cross-attention mechanism can include: using image features as the query, and a condition vector composed of defect annotation information and viewpoint feature information as the key and value. By calculating the similarity between the query and key, an attention weight is obtained. This attention weight is then used to weight and sum the values, thereby effectively injecting conditional information into the image features. Feature fusion processing allows the defect image generation model to focus on the guidance of external conditions with certain weights during denoising, generating defects that meet the requirements.

[0049] The decoder is used to upsample the processed feature map to obtain the defect image corresponding to the input image. Specifically, the decoder can be the upsampling part of the U-Net network that is symmetrical to the encoder, consisting of multiple upsampling layers (such as transposed convolutional or interpolation layers) and convolutional layers. The decoder receives the "processed feature map" fused by the attention module and upsamples it step by step to restore the spatial resolution of the image (e.g., upsampling from 64x64 back to 512x512).

[0050] Specifically, each layer of the decoder can also receive feature maps from the corresponding layer of the encoder through skip connections. This helps to restore resolution while preserving more low-level detail information, thus generating a clearer and more realistic defect image. The final output of the decoder is a generated image of the same size as the input image, which is the desired defect image.

[0051] Taking the diffusion model as an example, the working process of the defect image generation model is explained as follows: In the forward diffusion process, noise is gradually added to the target defect image (the ground truth during training) until it becomes completely Gaussian noise. In the backward denoising process, the diffusion model (containing an encoder, an attention module, and a decoder) is used to predict the noise added at each step. The input to the defect image generation model can include: the noisy image (or features in the latent space), the temporal step embedding indicating the current denoising step number, and conditional information: namely, the defect annotation information and viewpoint feature information introduced through the attention module. This defect image generation model extracts features from the noisy image through the encoder, fuses the conditional information with the image features through the attention module, and finally outputs the predicted noise through the decoder. After multiple iterative denoising steps, a clear, high-quality defect image is finally obtained.

[0052] This application's embodiments achieve more accurate and controllable defect image sample generation by employing a diffusion model architecture with a specific attention module. The encoder-decoder structure ensures the quality and resolution of the generated images, while the attention module introducing a cross-attention mechanism ensures that the generated defects strictly adhere to the user's constraints on location (via mask), type (via text), and viewpoint characteristics (via viewpoint features), thus providing a reliable foundation for multi-viewpoint consistent generation.

[0053] In some embodiments, the process of obtaining viewpoint feature information of the input image includes:

[0054] For each candidate viewpoint, image enhancement is performed on the sample defect image under the current candidate viewpoint to obtain the positive sample set of defect images corresponding to the current candidate viewpoint;

[0055] Based on the sample defect images from other candidate viewpoints besides the current candidate viewpoint, construct the negative sample set of defect images corresponding to the current candidate viewpoint.

[0056] The view feature extraction model corresponding to the current candidate viewpoint is trained based on the positive sample set and the negative sample set of defect images to obtain the trained view feature extraction model; the trained view feature extraction model is used to extract the view feature information of the input image of the current candidate viewpoint.

[0057] The training objectives of the viewpoint feature extraction model include: for any sample defect image under the current candidate viewpoint, increasing the similarity between the sample defect image and each positive sample of defect images in the positive sample set of defect images, and / or decreasing the similarity between the sample defect image and each negative sample of defect images in the negative sample set of defect images.

[0058] Among them, positive defect image samples refer to samples that are essentially of the same class (referring to the same viewpoint) but have undergone different transformations. The defect image generation model needs to learn that they are similar.

[0059] Specifically, for each candidate viewpoint (e.g., viewpoints A, B, and C), all sample defect images from that viewpoint are extracted from the aforementioned multi-viewpoint dataset. Image enhancement may involve applying a series of random image transformations to each sample defect image, preserving the semantic content, to create multiple variants. Transformations may include: 1. Geometric transformations: random horizontal or vertical flipping, small random rotation, small random cropping; 2. Color transformations: random adjustment of brightness, contrast, and saturation; 3. Noise injection: adding Gaussian noise or salt-and-pepper noise.

[0060] For the original image I_original, two enhanced versions Aug1(I_original) and Aug2(I_original) are obtained through two different random enhancements. The images (Aug1(I_original), Aug2(I_original)) then constitute a positive sample pair. All such image pairs constitute the set of positive defect images for that viewpoint. The model will learn to bring different enhanced versions of the same image closer together in the feature space.

[0061] Correspondingly, defect image negative samples refer to samples belonging to different classes (here, different viewpoints), and the model needs to learn that they are dissimilar. Specifically, for the current viewpoint (e.g., viewpoint A), all sample defect images are extracted from other candidate viewpoints (i.e., viewpoints B and C). An original image I_A is taken from viewpoint A, and an image I_B (or I_C) is arbitrarily taken from viewpoint B or viewpoint C. This pair of images (I_A, I_B) constitutes a defect image negative sample pair. All such image pairs composed of images from different viewpoints constitute the defect image negative sample set corresponding to viewpoint A. The model will learn to project images from different viewpoints into the feature space.

[0062] The view feature extraction model corresponding to the current candidate view is trained based on the positive sample set and the negative sample set of defect images, so as to obtain a model that can extract features with strong view discriminative power.

[0063] Specifically, the backbone network of the viewpoint feature extraction model is preferably a Vision Transformer (ViT) or a deep convolutional neural network (such as ResNet). The role of this network is to map the input image into a high-dimensional, normalized feature vector. Taking a batch of N samples as an example, the training process is explained as follows: First, a training batch is constructed: an anchor sample x is randomly selected (e.g., an enhanced image from viewpoint A). From the set of positive defect images corresponding to candidate viewpoint A, another enhanced version k+ corresponding to x is selected as the only positive sample, and from the set of negative defect images corresponding to candidate viewpoint A, N-1 images {k_i | i=1,2,...,N-1} from other viewpoints (B or C) are randomly selected as negative samples. The model forward propagation process: the anchor sample x, the positive sample k+, and all N-1 negative samples k_i are input into the ViT model to obtain the corresponding feature vectors q, k+, and {k_i}. The InfoNCE (noise contrast estimation) loss function can be used, and its formula is as follows:

[0064] .

[0065] Where q·k+ represents the dot product of anchor feature q and positive sample feature k+ (i.e., cosine similarity, since the features have been normalized); q·k_i represents the dot product of anchor feature q and negative sample feature k_i; τ is a temperature hyperparameter used to adjust the degree of attention given to difficult negative samples.

[0066] The loss function aims to maximize the similarity (q·k+) between the anchor point and positive samples, while minimizing the similarity (q·k_i) between the anchor point and all negative samples. By optimizing this loss, the model is motivated to learn image feature representations that can keenly distinguish between different viewpoints.

[0067] After training, the viewpoint feature extraction model can be used to extract viewpoint feature information from any input image. For example, during the training or inference of the diffusion model in the aforementioned steps, when processing an image from a specific viewpoint (e.g., viewpoint B), regardless of whether the image is OK or NG, it is first input into the pre-trained viewpoint feature extraction model corresponding to viewpoint B. The feature vector output by this model (or the result of further projection of this vector), as viewpoint feature information, is fed into the attention module of the diffusion model, serving as conditional guidance along with defect annotation information to ensure that the generated defects conform to the observation characteristics of that viewpoint.

[0068] By introducing a viewpoint feature extraction model based on contrastive learning, the embodiments of this application can automatically and efficiently learn subtle and crucial feature differences between different viewpoints. This learned viewpoint feature information serves as a powerful guiding signal, injected into the diffusion model, enabling the generator to "aware" which viewpoint it is generating the image for. This ensures that the generated defects are highly consistent with the real image of the target viewpoint in terms of texture, lighting, and perspective distortion, greatly improving the overall consistency and realism of multi-viewpoint generation.

[0069] In some embodiments, the encoder includes at least one self-attention layer; the self-attention layer is used to divide the input image into multiple sub-regions and perform self-attention calculations on the images within each sub-region to obtain the image features of the input image;

[0070] Multiple sub-regions are obtained by dividing the input image based on the region size; the region size of the sub-region is determined based on the image size of the input image; wherein, the region size is positively correlated with the image size, and the ratio of the region size to the image size satisfies a preset condition.

[0071] In the traditional U-Net architecture, the self-attention layer typically performs global self-attention computation, meaning each pixel in the feature map needs to be similar to all other pixels in the feature map. For example, when the feature map size is H x W, the computational complexity is O((H x W)²), which is computationally expensive and consumes a lot of memory for generating high-resolution industrial images (e.g., 1024x1024 and above). Therefore, in this embodiment, the self-attention layer in the encoder is optimized by replacing global self-attention with local self-attention. The local self-attention mechanism utilizes the locality prior of features in high-resolution images. That is, the correlation between a pixel and its neighboring pixels is usually much higher than its correlation with distant pixels. By restricting global computation to a local window, the computational load can be significantly reduced without much loss of generation quality.

[0072] Specifically, for a feature map of size H × W × C input to the self-attention layer, it is uniformly divided into multiple non-overlapping sub-regions (windows) of size M × M. For example, a 64 × 64 feature map can be divided into 64 8 × 8 sub-regions. Each sub-region contains M × M pixels (64 in this example).

[0073] Within each sub-region, standard self-attention computation is performed independently. Specifically, each sub-region is treated as an independent sequence. All pixels (M × M) within the sub-region are linearly projected to generate Query (Q), Key (K), and Value (V) matrices. The dot product of Q and K within the sub-region is calculated, and after softmax, an attention weight matrix is ​​obtained. This weight matrix has a size of (M × M) × (M × M). The obtained attention weights are used to weight and sum V, outputting a new feature representation for the sub-region. After computation for all sub-regions, they are reassembled according to their original spatial positions to form a complete output feature map enhanced by local self-attention.

[0074] By introducing locality priors that conform to the characteristics of image data, the local attention mechanism significantly reduces computational load while effectively preserving key feature information, minimizing its impact on the quality of subsequent defect generation. This enables the defect image generation model of this application to efficiently process high-resolution industrial images, meeting the stringent requirements for image detail in industrial defect detection and solving the core bottleneck problem of slow inference speed and high resource consumption of the original Stable Diffusion model at high resolutions. By implementing local window calculation in the encoder's self-attention layer, this application achieves an order-of-magnitude improvement in computational efficiency while maintaining the quality of the generated image, making the multi-view high-resolution defect image generation method feasible for industrial applications.

[0075] Correspondingly, while a fixed window size (e.g., 8x8) may be effective for the resolutions seen during model training, it may not be optimal when the model needs to generalize to higher or lower inference resolutions. For example, a receptive field that is effective in an 8x8 window will have a relatively smaller receptive range when directly applied to a feature map with doubled resolution, and may not be able to capture enough contextual information.

[0076] Therefore, in this embodiment, a dynamic window size mechanism is introduced to dynamically determine the window size based on the input size. This aims to establish an adaptive relationship between the image size and the window size, ensuring that local attention can always maintain a relatively stable perceptual range that matches the scale of the image content.

[0077] Specifically, the input image size can be the size (H, W) of the feature map to be used for local self-attention calculation. A fixed scaling factor r can be set (e.g., r = 1 / 8). The region size M is determined by the formula M = ceil(r * H) or M = ceil(r * W). To ensure the window is square, the smaller of H and W is usually used for calculation, or the values ​​are calculated separately and then averaged. For example, for a feature map with H=W=64, M = ceil(64 / 8) = 8; for a feature map with H=W=128, M = ceil(128 / 8) = 16. This ensures that the relative receptive field (the proportion relative to the entire image) covered by each window remains constant regardless of the feature map resolution. Specifically, a size-window mapping table can also be predefined to determine the region size of the sub-regions. For example, the mapping table can specify that if H <= 64, then M = 8. If 64 < H <= 128, then set M = 16. If 128 < H <= 256, then set M = 32, and so on. This discretization avoids frequent small changes in the window size, ensuring adaptability while facilitating engineering implementation and computational optimization.

[0078] After dynamically determining the region size M, the input image is divided into multiple sub-regions based on the region size. That is, the H × W feature map is divided into multiple M × M non-overlapping windows (if not divisible, the edge parts can be padded).

[0079] This application's embodiments introduce a dynamic window mechanism, enabling the model to automatically expand the absolute size of the local window when encountering higher-resolution images not seen during training. This maintains a relative receptive field similar to that during training, effectively avoiding insufficient contextual information capture due to resolution increases. Simultaneously, a smaller window size is used for lower-resolution feature maps, avoiding unnecessary computational overhead; a larger window size is used for high-resolution feature maps, acquiring richer contextual information with an acceptable computational increment, achieving an adaptive balance between computational efficiency and generation quality. Furthermore, at high resolutions, excessively small windows can cause the model to lack global coordination capabilities, easily leading to repetitive defect patterns or textures in different parts of the generated image. This scheme, by dynamically increasing the window size, provides the model with a wider field of view, enabling it to better coordinate the overall image generation structure, thereby effectively suppressing feature duplication.

[0080] In some embodiments, the image generation model includes a diffusion model; the diffusion model further includes a resolution-aware downsampler configured at the input of the encoder and a resolution-aware upsampler configured at the output.

[0081] Among them, the resolution-aware downsampler is used to downsample the input image to the target size according to a preset downsampling factor;

[0082] The resolution-aware upsampler is used to upsample the defect image corresponding to the input image to the target resolution according to a preset upsampling factor; the upsampling factor is determined based on the downsampling factor.

[0083] Key enhancements were made to the input and output of the diffusion model (especially its U-Net structure). By introducing a resolution-aware sampler, arbitrary input resolutions were adapted to the optimal size familiar during model training, thereby ensuring the quality of the generated data.

[0084] Specifically, when generating images at resolutions significantly higher than the training resolution (e.g., the model is trained at 256x256 resolution but needs to generate 1024x1024 images for inference), direct inference can lead to repetitive patterns in deep features of UNet, manifesting as multiple identical defects or textures being generated in the image. This is because high-resolution feature maps still maintain a large spatial size in the deep layers (bottleneck) of UNet, causing the self-attention mechanism to capture abnormal long-range dependencies.

[0085] To address this issue, embodiments of this application introduce a resolution-aware downsampler (RAD) and a resolution-aware upsampler (RAU) into the traditional diffusion model's U-Net structure. The resolution-aware downsampler is used to quickly downsample the high-resolution input image (or latent features) to a stable target size compatible with the model's training, based on a preset downsampling factor α. Specifically, the RAD is essentially a parameterized convolutional layer whose stride is set to be equal to the downsampling factor α. The formula is defined as follows: ;

[0086] Where x: input feature map; C: convolution operation; k: target convolution kernel size; p: target padding parameter (e.g., 1, to ensure the integrity of size calculation); α: target convolution stride, i.e. downsampling factor (e.g., to downsample 1024 to 256, then α=4); d: target dilation rate (e.g., 2, to increase the receptive field without increasing the number of parameters through dilated convolution).

[0087] For example, to downsample a 1024x1024 input image by a factor of 4 to 256x256, the RAD layer can be set to k=3, p=1, α=4, d=2. This layer performs a convolution operation with a kernel of 3, padding of 1, stride of 4, and dilation of 2, directly outputting a 256x256 feature map.

[0088] Correspondingly, the resolution-aware upsampler is used to upsample the low-resolution feature map output by the decoder back to the original target resolution according to a preset upsampling factor β. The upsampling factor β is determined based on the downsampling factor α, and is usually β = α. For example, if the input is downsampled by a factor of 4, the output needs to be upsampled by a factor of 4 to restore the original size.

[0089] Specifically, a RAU can consist of an interpolation operation and a convolutional smoothing operation in sequence. The formula is defined as follows: ;

[0090] Where `interp(x, β)`: interpolates and amplifies the input feature map `x` by a magnification factor of β (e.g., nearest neighbor interpolation or bilinear interpolation), resulting in an amplified feature map. This operation restores the size but does not add details. `C_{3,1,1,1}`: represents a standard 3x3 convolution with parameters: kernel size = 3, padding = 1, stride = 1, and dilation rate = 1. This convolutional layer smooths and fine-tunes the details of the slightly blurred feature map after interpolation, improving the visual quality of the output image. Continuing the previous example, the decoder outputs a 256x256 feature map. The RAU first upsamples it by a factor of 4 to 1024x1024 through interpolation, then performs a 3x3 convolution for smoothing, finally obtaining the output image at the target resolution.

[0091] This embodiment utilizes RAD to reduce the high-resolution input to a size familiar to the model at the entry point, ensuring that the subsequent UNet backbone (especially the self-attention layer) always operates within its optimal, trained size range. This avoids the self-attention mechanism malfunction and feature duplication issues caused by excessively large feature maps. Considering that the core denoising process for defect image generation is performed at a familiar and stable resolution, the realism of the generated defect's shape and texture is guaranteed. RAU is responsible for upscaling the high-quality low-resolution results to the target size, significantly reducing computational cost compared to processing high-resolution features throughout the entire UNet. Because the most time-consuming UNet backbone network processes downsampled low-resolution features, the inference process is greatly accelerated. Furthermore, this embodiment does not require retraining the entire UNet; only fine-tuning or direct replacement of the input and output layers is needed to enable existing models to process high-resolution images, demonstrating strong versatility and feasibility.

[0092] In some embodiments, downsampling the input image to a target size according to a preset downsampling factor includes:

[0093] The convolution parameters are determined based on a preset downsampling factor. The convolution parameters include at least one of the following: target convolution stride, target convolution kernel size, target padding parameter, and target dilation rate. The target convolution stride is equal to the downsampling factor.

[0094] The input image is convolved according to the convolution parameters to obtain the input image of the target size.

[0095] The convolution parameters can be dynamically configured based on the downsampling factor to ensure accurate and efficient downsampling of the input feature map to the target size through a single-step convolution operation, while preserving as much important feature information as possible. Specifically, the primary goal in selecting convolution parameters is to ensure that the output feature map size is 1 / α of the input size. These parameters can include: Target convolution stride: This is the most critical parameter. The target convolution stride is equal to the downsampling factor α. For example, to downsample by 4 times (α=4), the convolution stride is directly set to 4, thus achieving the target downsampling factor. The target convolution kernel size k needs to match the stride α to ensure sufficient and effective sampling. It can be set to an odd number greater than 1, such as 3. Larger convolution kernels (such as 5) can provide a larger receptive field but increase computational cost. Target padding parameters, such as p, can be used to control the size of the output feature map. To ensure the accuracy of size calculation and avoid information loss at boundaries, it can be set based on the convolution kernel size and stride. For example, it can be set to p = (k - 1) / 2 (when stride = 1). Considering that during large-stride downsampling, the goal is usually to ensure that the output size is strictly H_out = H_in / α and W_out = W_in / α, padding is usually set to 0 or 1, which needs to be inversely calculated based on the specific size formula. Convolution parameters can also include the target dilation rate: the dilation rate d can be used to introduce dilated convolutions, which can significantly increase the receptive field without increasing the number of parameters. For example, when α=4, k=3, and d=2, its effective receptive field is equivalent to a standard 5x5 convolution. This helps to capture a sufficient range of contextual information during aggressive downsampling, avoiding excessive feature loss.

[0096] After determining the above quadruple convolution parameters (k, p, α, d), a convolutional layer is constructed using these parameters, and a convolution operation is performed on the input feature map x, thereby obtaining the downsampled feature map, the size of which is determined by the standard convolution output size formula: By carefully selecting the parameters, it can be ensured that H_out is exactly equal to H_in / α.

[0097] For example, assuming the input image size is 1024x1024, and the preset downsampling factor α=4, the goal is to reduce the feature map size to 256x256. A feasible parameter configuration is: k=3, p=1, α=4, d=2; the calculation and verification are as follows: (In the actual framework, division will be rounded down, which can be precisely controlled by fine-tuning the padding.) It can be seen that this configuration uses a 3x3 dilated convolution with a stride of 4 to downsample 1024x1024 to 256x256 in one step. At the same time, its effective receptive field is large, which is beneficial to preserving the overall structural information of the image.

[0098] This application's embodiments, through an adaptive convolution parameter determination method, can accurately downsample any input resolution to a preset target size, providing a stable input environment for the subsequent UNet core processing modules. Specifically, by introducing techniques such as dilated convolution, the receptive field is expanded as much as possible while aggressively downsampling, preserving richer global semantic information and laying the foundation for generating high-quality, structurally sound images.

[0099] In some embodiments, the defect image corresponding to the input image is upsampled to the target resolution according to a preset upsampling factor, including:

[0100] Using a preset upsampling factor as the magnification factor, the defect image corresponding to the input image is interpolated and magnified to obtain the magnified feature map;

[0101] A convolutional smoothing operation is performed on the magnified feature map to obtain the defect image corresponding to the input image at the target resolution.

[0102] In this process, the low-resolution feature map output by the UNet decoder is reconstructed and upsampled to the final target resolution with high quality. Specifically, the defect image corresponding to the input image refers to the low-resolution feature map output by the decoder that has not yet undergone final upsampling. For example, a feature map with a size of 256x256. An interpolation algorithm is used to enlarge the input feature map by a factor of β. The upsampling factor β is determined based on the downsampling factor α in the previous step, and is usually β=α. Specific interpolation methods include: 1. Nearest neighbor interpolation: the fastest in computation, but may produce blocky artifacts; 2. Bilinear interpolation: achieves a good balance between computational speed and smoothness, and is the preferred implementation; 3. Bicubic interpolation: produces smoother edges, but has a slightly higher computational cost.

[0103] After interpolation, a magnified feature map is obtained, enlarged by a factor of β. For example, a 256x256 feature map is upsampled by a factor of 4 using bilinear interpolation to obtain a magnified feature map of 1024x1024. It should be noted that while this feature map is the correct size, it often appears blurry and lacks high-frequency details. Therefore, convolutional smoothing is then performed on the magnified feature map to obtain a defect image corresponding to the input image at the target resolution. This repairs and refines the blur and artifacts introduced by the interpolation operation, restoring the image's clarity and detail to meet the quality requirements of industrial vision.

[0104] A standard convolutional layer can be used for convolutional smoothing. A preferred configuration is C_{3,1,1,1}, where the kernel size k=3, padding p=1, stride s=1, and dilation d=1. This 3x3 convolutional layer with a stride of 1 acts like a smart smoothing filter. It operates within each local 3x3 region of the magnified feature map, learning how to reconstruct sharp edges and textures from the blurred interpolation results. After processing by this convolutional layer, a sharp image with enhanced details at the target resolution is finally obtained, which corresponds to the defect image of the input image.

[0105] The interpolation operation used in this embodiment has extremely low computational cost, and the subsequent 3x3 convolution also has a small computational cost, thus achieving efficient upsampling, which is faster than using transposed convolution or pixel shuffling. The interpolation operation ensures the accuracy of spatial dimension expansion, while the subsequent learnable convolutional layer is responsible for "refinement." It can learn how to supplement and sharpen image details based on training data, thereby generating a final image that is much clearer than simple interpolation and more natural than other upsampling methods. Compared with the checkerboard artifacts that sometimes occur with transposed convolution, this embodiment effectively avoids such problems by combining interpolation and convolution, resulting in a smoother and more natural output.

[0106] In some embodiments, after training an image generation model with sample defect images of the same object from a first perspective as input and sample defect images from a second perspective as output to obtain a second defect image generation model corresponding to the second perspective, the method further includes:

[0107] Obtain a defect-free image of the target object from the anchor point viewpoint, along with the defect location annotations corresponding to the anchor point viewpoint; the anchor point viewpoint can be any one of multiple candidate viewpoints.

[0108] The defect-free image and defect location annotations from the anchor point view are input into the defect image generation model corresponding to the anchor point view to obtain the defect image of the target object from the anchor point view.

[0109] The defect image of the target object from the anchor point view and the defect-free image of the target object from the target view are input into the defect image generation model corresponding to the target view to obtain the defect image of the target object from the target view; wherein, the target view is any one of multiple candidate views other than the anchor point view.

[0110] The target object can be a completely new object that has not appeared in the training set. For example, a defect-free PCB board that has just come off the production line. The anchor point viewpoint can be consistent with the first viewpoint selected during training, which is viewpoint A in this embodiment. The defect-free image refers to the actual defect-free image (OK image) of the target object under viewpoint A, obtained through an image acquisition system.

[0111] Defect location markers can be user-specified or generated using a random / heuristic algorithm, indicating where defects are desired to be generated. They can exist as a binary mask, where areas with a value of 1 represent the desired defect location, and 0 represents a non-defect area. For example, a user can draw a white square at a solder joint location on a PCB board as a defect location marker.

[0112] The acquired defect-free image from viewpoint A and the defect location mask are input together into the pre-trained model G_A. This allows model G_A to generate a realistic defect at a specified location based on the background and structure of the defect-free image, guided by the defect location mask. This results in the first generated image—the defect image of the target object from the anchor point viewpoint (viewpoint A). This image forms the basis for all subsequent generation; the quality and location of the defect are crucial.

[0113] This leads to the second stage of cascading the generation of multi-view defect images. The target viewpoint is any one of several candidate viewpoints, excluding the anchor point viewpoint, such as viewpoint B or viewpoint C. A defect-free image of the target object is obtained from the target viewpoint (e.g., viewpoint B). The previously obtained defect image from viewpoint A and the defect-free image from viewpoint B are input together into the trained model G_B. Model G_B has learned the mapping from the defect representation in viewpoint A to the defect representation in viewpoint B. The model receives the defect image from viewpoint A to obtain defect information, and simultaneously receives the defect-free image from viewpoint B to obtain background and structural information for viewpoint B. At the corresponding position in the defect-free image of viewpoint B, a defect that morphologically matches the defect in viewpoint A is generated, thus obtaining the defect image of the target object from the target viewpoint (viewpoint B). Similarly, to generate a defect image from viewpoint C, the process of the second stage is repeated, inputting the defect image from viewpoint A and the defect-free image from viewpoint C into model G_C. Since models G_B and G_C are both generated from the same defect image from viewpoint A, the three defect images of the target object obtained from viewpoints A, B, and C are naturally aligned and consistent in terms of physical location and macroscopic shape, thus solving the inconsistency problem caused by multiple models being generated independently.

[0114] This application provides a complete automated generation pipeline from "defect-free OK images" to "consistent multi-view NG images". It only requires collecting a set of multi-view defect-free images of the target object to generate a large amount of multi-view defect data with consistent annotations in batches, which greatly reduces the cost of data collection and annotation. Furthermore, by introducing the ability for users to easily control the location of generated defects by specifying different defect location masks, it meets various simulation and testing needs.

[0115] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0116] Based on the same inventive concept, this application also provides an image generation apparatus. The solution provided by this apparatus is similar to the solution described in the above method. Therefore, the specific limitations of one or more image generation apparatus embodiments provided below can be found in the limitations of the image generation method above, and will not be repeated here.

[0117] like Figure 3 As shown, this application embodiment provides an image generation apparatus 300, including:

[0118] The acquisition module 302 is used to acquire sample defect images of the same object from multiple candidate viewpoints.

[0119] The first training module 304 is used to train a preset image generation model with sample defect-free images of the same object from a first perspective as input and sample defect images from the first perspective as output, to obtain a first defect image generation model corresponding to the first perspective; the first defect image generation model is used to generate defect images from the first perspective based on the defect-free images from the first perspective; the first perspective is any one of multiple candidate perspectives.

[0120] The second training module 306 is used to train the image generation model with sample defect images of the same object under the first viewpoint as input and sample defect images under the second viewpoint as output, so as to obtain the second defect image generation model corresponding to the second viewpoint; the second defect image generation model is used to generate defect images under the second viewpoint based on the defect-free images under the first viewpoint; the second viewpoint is any one of multiple candidate viewpoints other than the first viewpoint.

[0121] In some embodiments, the image generation model includes a diffusion model; the diffusion model includes an encoder, an attention module, and a decoder connected in sequence.

[0122] The encoder is used to downsample the input image to obtain the image features of the input image; the attention module is used to fuse the defect annotation information, viewpoint feature information and image features of the input image based on the cross-attention mechanism to obtain the processed feature map.

[0123] The decoder is used to upsample the processed feature map to obtain the defect image corresponding to the input image; wherein, the defect annotation information includes at least one of the defect location annotation and defect type annotation of the input image.

[0124] In some embodiments, in terms of acquiring viewpoint feature information of the input image, the first training module 304 is specifically used for:

[0125] For each candidate viewpoint, image enhancement is performed on the sample defect image under the current candidate viewpoint to obtain the positive sample set of defect images corresponding to the current candidate viewpoint;

[0126] Based on the sample defect images from other candidate viewpoints besides the current candidate viewpoint, construct the negative sample set of defect images corresponding to the current candidate viewpoint.

[0127] The view feature extraction model corresponding to the current candidate viewpoint is trained based on the positive sample set and the negative sample set of defect images to obtain the trained view feature extraction model; the trained view feature extraction model is used to extract the view feature information of the input image of the current candidate viewpoint.

[0128] The training objectives of the viewpoint feature extraction model include: for any sample defect image under the current candidate viewpoint, increasing the similarity between the sample defect image and each positive sample of defect images in the positive sample set of defect images, and / or decreasing the similarity between the sample defect image and each negative sample of defect images in the negative sample set of defect images.

[0129] In some embodiments, the encoder includes at least one self-attention layer; the self-attention layer is used to divide the input image into multiple sub-regions and perform self-attention calculations on the images within each sub-region to obtain image features of the input image;

[0130] Multiple sub-regions are obtained by dividing the input image based on the region size; the region size of the sub-region is determined based on the image size of the input image; wherein, the region size is positively correlated with the image size, and the ratio of the region size to the image size satisfies a preset condition.

[0131] In some embodiments, the image generation model includes a diffusion model; the diffusion model further includes a resolution-aware downsampler configured at the input of the encoder and a resolution-aware upsampler configured at the output.

[0132] Among them, the resolution-aware downsampler is used to downsample the input image to the target size according to a preset downsampling factor;

[0133] The resolution-aware upsampler is used to upsample the defect image corresponding to the input image to the target resolution according to a preset upsampling factor; the upsampling factor is determined based on the downsampling factor.

[0134] In some embodiments, the first training module 304 is specifically used for downsampling the input image to a target size according to a preset downsampling factor:

[0135] The convolution parameters are determined based on a preset downsampling factor. The convolution parameters include at least one of the following: target convolution stride, target convolution kernel size, target padding parameter, and target dilation rate. The target convolution stride is equal to the downsampling factor.

[0136] The input image is convolved according to the convolution parameters to obtain the input image of the target size.

[0137] In some embodiments, the first training module 304 is specifically used for upsampling the defect image corresponding to the input image to the target resolution according to a preset upsampling factor:

[0138] Using a preset upsampling factor as the magnification factor, the defect image corresponding to the input image is interpolated and magnified to obtain the magnified feature map;

[0139] A convolutional smoothing operation is performed on the magnified feature map to obtain the defect image corresponding to the input image at the target resolution.

[0140] In some embodiments, the first training module 304 is further configured to:

[0141] Obtain a defect-free image of the target object from the anchor point view and the defect location annotation corresponding to the anchor point view; the anchor point view can be any one of multiple candidate views; input the defect-free image and defect location annotation from the anchor point view into the defect image generation model corresponding to the anchor point view to obtain the defect image of the target object from the anchor point view; input the defect image of the target object from the anchor point view and the defect-free image of the target object from the target view into the defect image generation model corresponding to the target view to obtain the defect image of the target object from the target view; wherein, the target view can be any one of multiple candidate views other than the anchor point view.

[0142] Each module in the aforementioned image generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0143] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to the image generation method. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps of the image generation method described above.

[0144] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it performs the steps in the image generation method described above. The display unit of the computer device is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen; the input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs or touchpads set on the casing of the computer device, or external keyboards, touchpads or mice, etc.

[0145] Those skilled in the art will understand that Figure 4 or Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0146] In some embodiments, a computer device is provided, the computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0147] In some embodiments, such as Figure 6 The diagram shows the internal structure of a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the above-described method embodiments.

[0148] In some embodiments, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0149] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0150] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0151] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image generation method, characterized in that, include: Obtain sample defect images of the same object from multiple candidate viewpoints; Using defect-free sample images of the same object from a first perspective as input and defective sample images from the first perspective as output, a preset image generation model is trained to obtain a first defective image generation model corresponding to the first perspective. The first defective image generation model is used to generate a defective image from the first perspective based on the defect-free image from the first perspective. The first perspective is any one of the multiple candidate perspectives. The image generation model includes a diffusion model. The diffusion model includes an encoder, an attention module, and a decoder connected in sequence. The attention module is used to fuse the defect annotation information of the input image of the encoder, the perspective feature information of the input image, and the image features based on a cross-attention mechanism. After multiple iterative denoising steps, the defective image corresponding to the input image is obtained. The image generation model is trained using sample defect images of the same object from the first perspective as input and sample defect images from the second perspective as output to obtain a second defect image generation model corresponding to the second perspective; the second defect image generation model is used to generate defect images from the second perspective based on the defect-free images from the first perspective; the second perspective is any one of the multiple candidate perspectives other than the first perspective.

2. The method according to claim 1, characterized in that, The encoder is used to downsample the input image to obtain the image features of the input image; the attention module is used to fuse the defect annotation information of the input image, the viewpoint feature information of the input image, and the image features to obtain a processed feature map. The decoder is used to upsample the processed feature map to obtain a defect image corresponding to the input image; wherein, the defect annotation information includes at least one of the defect location annotation and defect type annotation of the input image.

3. The method according to claim 2, characterized in that, The process of obtaining the viewpoint feature information of the input image includes: For each of the candidate viewpoints, image enhancement is performed on the sample defect image under the current candidate viewpoint to obtain the positive sample set of defect images corresponding to the current candidate viewpoint; Based on the sample defect images from the candidate viewpoints other than the current candidate viewpoint, construct a negative sample set of defect images corresponding to the current candidate viewpoint. The view feature extraction model corresponding to the current candidate viewpoint is trained based on the positive sample set and the negative sample set of the defect images to obtain the trained view feature extraction model; the trained view feature extraction model is used to extract view feature information of the input image of the current candidate viewpoint. The training objectives of the viewpoint feature extraction model include: for any sample defect image under the current candidate viewpoint, increasing the similarity between the sample defect image and each positive sample of defect images in the positive sample set of defect images, and / or decreasing the similarity between the sample defect image and each negative sample of defect images in the negative sample set of defect images.

4. The method according to claim 2, characterized in that, The encoder includes at least one self-attention layer; the self-attention layer is used to divide the input image into multiple sub-regions, and perform self-attention calculation on the image within each sub-region to obtain the image features of the input image; The multiple sub-regions are obtained by dividing the input image based on the region size; the region size of the sub-region is determined based on the image size of the input image; wherein, the region size is positively correlated with the image size, and the ratio of the region size to the image size satisfies a preset condition.

5. The method according to any one of claims 1-4, characterized in that, The image generation model includes a diffusion model; the diffusion model further includes a resolution-aware downsampler configured at the input end of the encoder and a resolution-aware upsampler configured at the output end. The resolution-aware downsampler is used to downsample the input image to a target size according to a preset downsampling factor. The resolution-aware upsampler is used to upsample the defect image corresponding to the input image to the target resolution according to a preset upsampling factor; the upsampling factor is determined based on the downsampling factor.

6. The method according to claim 5, characterized in that, The step of downsampling the input image to the target size according to a preset downsampling factor includes: The convolution parameters are determined based on a preset downsampling factor; the convolution parameters include at least one of the target convolution stride, target convolution kernel size, target padding parameter, and target dilation rate; wherein the target convolution stride is equal to the downsampling factor; The input image is convolved according to the convolution parameters to obtain the input image of the target size.

7. The method according to claim 5, characterized in that, The step of upsampling the defect image corresponding to the input image to the target resolution according to a preset upsampling factor includes: Using a preset upsampling factor as the magnification factor, the defect image corresponding to the input image is interpolated and magnified to obtain the magnified feature map; A convolutional smoothing operation is performed on the magnified feature map to obtain the defect image corresponding to the input image at the target resolution.

8. The method according to claim 1, characterized in that, After training the image generation model using sample defect images of the same object from the first viewpoint as input and sample defect images from the second viewpoint as output to obtain a second defect image generation model corresponding to the second viewpoint, the method further includes: Obtain a defect-free image of the target object from the anchor point view and the defect location annotation corresponding to the anchor point view; the anchor point view is any one of the multiple candidate views. The defect-free image and the defect location annotation under the anchor point view are input into the defect image generation model corresponding to the anchor point view to obtain the defect image of the target object under the anchor point view. The defect image of the target object under the anchor point view and the defect-free image of the target object under the target view are input into the defect image generation model corresponding to the target view to obtain the defect image of the target object under the target view; wherein, the target view is any one of the plurality of candidate views other than the anchor point view.

9. An image generation apparatus, characterized in that, include: The acquisition module is used to acquire sample defect images of the same object from multiple candidate viewpoints. The first training module is used to train a preset image generation model with sample defect-free images of the same object from a first viewpoint as input and sample defect images from the first viewpoint as output, to obtain a first defect image generation model corresponding to the first viewpoint; the first defect image generation model is used to generate a defect image from the first viewpoint based on the defect-free image from the first viewpoint; the first viewpoint is any one of the multiple candidate viewpoints; the image generation model includes a diffusion model; the diffusion model includes an encoder, an attention module, and a decoder connected in sequence; the attention module is used to fuse the defect annotation information of the input image of the encoder, the viewpoint feature information of the input image, and the image features based on a cross-attention mechanism, and after multi-step iterative denoising, obtain the defect image corresponding to the input image; The second training module is used to train the image generation model with sample defect images of the same object under the first viewpoint as input and sample defect images under the second viewpoint as output, to obtain a second defect image generation model corresponding to the second viewpoint; the second defect image generation model is used to generate defect images under the second viewpoint based on the defect-free images under the first viewpoint; the second viewpoint is any one of the multiple candidate viewpoints other than the first viewpoint.

10. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images