Multi-subject defect image generation method based on position guidance

Abstract

The invention discloses a multi-subject defect image generation method based on position guidance. The method comprises the steps that a single-theme feature extractor and a multi-scale image denoising network based on position guidance are constructed, the single-theme feature extractor comprises a theme feature encoder and a theme layout generation module, and a reference image and a theme text are jointly encoded through a VAE image encoder, a CLIP text encoder and a theme feature extraction UNet network; the regional feature injection module is used for extracting high-expressivity image features and automatically generating spatial layout information of each theme, the regional feature injection module comprises a regional feature mask sub-module and a regional feature fusion sub-module, and isolation and directional expression of theme features are realized through explicit region division and a cross attention mechanism; a two-stage training mode is adopted, a single-theme feature extractor is pre-trained, and then an overall model is jointly trained. According to the method, the problems of topic feature confusion and unreasonable layout in a traditional method are effectively solved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a location-guided method for generating multi-topic defect images. Background Technology

[0002] With the development of deep learning technology, image generation technology has been widely used in fields such as intelligent manufacturing, defect detection, autonomous driving, and medical image analysis. Among existing image generation methods, single-topic image generation and multi-topic image generation are the two core tasks. Among them, multi-topic image generation requires the model to generate multiple regions with specific semantics or attributes in the same image, while maintaining the rationality of the spatial layout of these regions and the independence of their respective themes.

[0003] Existing multi-topic image generation methods mainly face the following problems: (1) Difficulty in maintaining subject identity: In multi-subject scenarios, existing methods often result in mutual interference or even confusion of different subject features, making it difficult to accurately reflect the features and boundaries of each subject in the synthesized image.

[0004] (2) Insufficient spatial relationship modeling: There are often specific spatial relationships (such as overlap, adjacency, etc.) between multiple subjects. However, existing methods are difficult to effectively model and control the spatial distribution and relative positional relationships between subjects, resulting in a large difference between the layout of the generated image and the real scene.

[0005] To address the aforementioned issues, there is an urgent need for a multi-theme image generation method that can comprehensively consider the identity, spatial distribution, and interrelationships of multiple themes, thereby enhancing the realism and application value of synthesized images. Summary of the Invention

[0006] The main objective of this invention is to provide a location-guided method for generating multi-topic defect images.

[0007] The second objective of this invention is to provide an electronic device.

[0008] A third objective of this invention is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above objectives, a first aspect of the present invention proposes a location-guided method for generating multi-topic defect images, comprising: A single-topic feature extractor for extracting topic features is constructed, including a topic feature encoder and a topic layout generation module; the topic feature encoder is used to jointly encode the input reference image and topic text to extract highly expressive image features of each topic; the topic layout generation module automatically generates spatial layout information of each topic based on the extracted topic features. A region feature injection module for extracting layout information is constructed, including a region feature masking submodule and a region feature fusion submodule; the region feature masking submodule performs explicit region division of latent space features according to the theme layout information; the region feature fusion submodule fuses the image features of each theme with the corresponding region features; A two-stage training approach is adopted: the first stage trains the single-topic feature extractor, and the second stage jointly trains the denoising UNet and the single-topic feature extractor. Based on the extracted thematic features and layout information, latent space variables are guided and denoised to generate high-quality multi-theme images.

[0010] Optionally, the topic feature encoder is specifically used for: During the coding phase, input the topic text. Compared with reference image The topic text is encoded using the CLIP text encoder to obtain the text embedding. The reference image is encoded using a VAE image encoder to obtain the image embedding. ; Spatial feature enhancement is achieved by extracting topic features from the spatial self-attention module within the UNet structure. This spatial self-attention module specifically comprises two residual modules, two Transformer blocks, and a downsampling unit. The first spatial self-attention module receives the reference image embedding. As input, where C is the number of feature channels, H is the height, and W is the width, the subsequent spatial self-attention module receives the image features enhanced by the previous spatial self-attention module as input; Topic feature extraction is achieved through the topic-text alignment module in UNet. The topic-text alignment module specifically includes two residual modules, two Transformer blocks, and an upsampling unit. The topic-text alignment module receives spatial self-attention image features from the spatial attention module at the same level, topic text features encoded by the CLIP text encoder, and topic image features extracted by the deep topic-text alignment module. The Transformer block uses a cross-attention mechanism to align the topic features with the topic image features.

[0011] Optionally, the theme layout generation module is specifically used for: The VAE decoder is used to decode the features of the subject image back into pixel space to obtain a mask image containing only the subject. The maximum 8-connected graph with non-zero pixel values ​​in the mask image is obtained by using a depth-first traversal algorithm, and the minimum bounding rectangle of the 8-connected graph is generated. Map the smallest bounding rectangle onto the coordinate system of the reference image to determine the specific location of the subject in the reference image; Overlay and merge the minimum bounding rectangles of multiple themes; The final topic location layout is output as a spatial prior for multi-topic image generation.

[0012] Optionally, the region feature mask submodule is specifically used for: Based on the input topic layout information, the latent space features are explicitly divided into regions. Using the binary mask matrix of the topic layout, each pixel of the latent space feature is multiplied with the mask value pixel by pixel to obtain multiple local feature maps.

[0013] Optionally, the regional feature fusion submodule is specifically used for: Efficient fusion between regional features and topic features is achieved through a cross-attention mechanism. Regional features are used as query vectors, and topic features are used as key vectors. The fused regional features are output using a residual connection method.

[0014] Optional two-stage training methods include: The first stage involves pre-training a single-topic feature extractor, using supervised learning to guide the encoder to extract features, calculating the mean squared error loss using the real features and the extracted features, and training the encoding model through backpropagation. The second stage involves joint training of the entire model, which combines the pre-trained single-topic feature extractor with the multi-scale region feature injection module, enabling the model to generate multi-topic images with corresponding layouts based on the extracted topic images and topic layouts.

[0015] To achieve the above objectives, a second aspect of this application provides an electronic device, including a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, for implementing the method described in the first aspect embodiment.

[0016] To achieve the above objectives, a third aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect.

[0017] The embodiments of this invention offer the following advantages: Addressing the difficulties in preserving subject identity and insufficient spatial relationship modeling in existing multi-topic image generation methods, a location-guided multi-topic defect image generation method is proposed. By introducing a single-topic feature extractor and a subject layout generation module, the separation of image features from different subjects is achieved, enhancing the independence of subject semantics. Furthermore, relying on a location-guided multi-scale image denoising network and a region feature injection mechanism, the spatial distribution of subject regions can be automatically established and precisely controlled, significantly enhancing the spatial rationality and subject distinguishability of the generated images, and avoiding the problems of subject feature confusion and unreasonable layout in traditional methods. In addition, the method of this invention reduces the dependence on large-scale manually labeled data, possesses strong adaptability and generalization ability, and can be widely applied to practical scenarios such as intelligent manufacturing and defect detection. Attached Figure Description

[0018] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is an architecture diagram of a location-guided multi-topic defect image generation method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the multi-topic location layout generation process in an embodiment of the present invention. Figure 3 This is a flowchart of the region feature injection module in an embodiment of the present invention. Detailed Implementation

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] To address the challenges of preserving subject identity and insufficient spatial relationship modeling in existing multi-topic image generation methods, this invention proposes a location-guided multi-topic defect image generation method. Specifically, to address the challenge of preserving subject identity, in the feature learning stage, a single-topic feature extractor is trained using mask reconstruction loss, enabling it to extract different subject image features based on subject text with a limited number of training samples. In the feature extraction stage, multiple single-topic feature extractors are used to extract different subject image features, avoiding early feature fusion. In the feature fusion stage, a multi-scale region feature injection module injects multiple subject features separately into an image denoising network, achieving subject feature isolation through region injection and preventing interference between subject features during fusion, thus better protecting subject identity. To address the challenge of insufficient spatial relationship modeling between subjects, this invention proposes a subject layout generation module. In the feature extraction stage, based on the distribution of defect subjects in the real reference image, a layout structure of subject regions is generated, providing correct and reasonable spatial layout information for subsequent multi-topic image generation.

[0022] The following describes a location-guided multi-topic defect image generation method according to an embodiment of the present invention, with reference to the accompanying drawings.

[0023] This embodiment provides a location-guided method for generating multi-topic defect images. For example... Figure 1 As shown, the method includes the following steps: S1, Construct a single-topic feature extractor for extracting topic features, including a topic feature encoder and a topic layout generation module; the topic feature encoder is used to jointly encode the input reference image and topic text to extract highly expressive image features of each topic; the topic layout generation module automatically generates spatial layout information of each topic based on the extracted topic features.

[0024] In this embodiment of the invention, the single-topic feature extractor comprises two sub-modules: a topic feature encoder and a topic layout generation module. The topic feature encoder jointly encodes the reference image and topic text. Trained using a mask reconstruction loss, it accurately extracts highly expressive image features of the corresponding topic with a limited number of samples, effectively avoiding early fusion and mutual interference between topic features and improving the separation of topic identities. The topic layout generation module, based on the features extracted by the topic feature encoder, reconstructs the mask image of each topic using a VAE image decoder. Through pixel analysis, minimum bounding rectangle extraction, and other steps, it automatically generates the spatial layout of each topic. Finally, the spatial location information of each topic is fused to form a reasonable topic region configuration, serving as a spatial prior for subsequent multi-topic image generation and providing precise layout guidance for subsequent feature injection and image synthesis.

[0025] Reference Figure 2 The following is a detailed description of each submodule.

[0026] (1) Topic feature encoder.

[0027] In this embodiment of the invention, the topic feature encoder mainly consists of three parts: a VAE image encoder for encoding the reference image, mapping the image from pixel space to a high-dimensional vector space; a CLIP text encoder for encoding the topic text description to be extracted, providing vector features for subsequent feature extraction; and a topic feature extraction UNet downsampling stage that feeds the reference image features into multiple spatial self-attention modules to identify the relationships between different regions of the image, understand the content and structure of the image, and dynamically assign attention scores to important regions of the image using a self-attention mechanism, thereby enhancing the image feature extraction effect. In the topic feature extraction UNet upsampling stage, a topic-text alignment module is used, employing a cross-attention mechanism to align the topic text with the topic features in the reference image.

[0028] Specifically, the workflow of a topic feature encoder can be described as follows: Encoding phase: Input topic text (SoI stands for Subject of Interest) and reference image (ref stands for reference) The CLIP text encoder encodes the topic text to obtain the text embedding. The VAE image encoder encodes the reference image to obtain the image embedding. Specifically, this invention uses Clip-Vit-Large-patch14 as the CLIP encoder.

[0029] Spatial Feature Enhancement Stage: Spatial feature enhancement is mainly achieved by the spatial self-attention module in the UNet structure for topic feature extraction. The spatial self-attention module specifically includes two residual modules, two Transformer blocks, and a downsampling unit. The first spatial self-attention module receives the reference image embedding. As input, where C is the number of feature channels, H is the height, and W is the width, the subsequent spatial self-attention module receives the image features enhanced by the preceding spatial self-attention module as input. Specifically, a feature map of size (C, H, W) is input to the spatial attention module. First, the feature map passes through the first residual module, and then... After the convolutional layer, the number of channels is doubled, outputting a feature map of size (2C, H, W). These feature maps then pass through a Transformer module for self-attention computation to extract high-level image semantic features. The feature map shape is then preserved to (2C, H, W) by a residual module. After passing through a second Transformer module, a downsampling module is used to downsample the feature map, reducing its spatial dimension, resulting in an output shape of... The feature map.

[0030] Topic Feature Extraction Stage: Topic feature extraction is primarily implemented by the topic-text alignment module in UNet. This module consists of two residual modules, two Transformer blocks, and an upsampling unit. The topic-text alignment module mainly receives spatial self-attention image features from the same-level spatial attention module, topic text features encoded by the CLIP text encoder, and topic image features extracted by the deep topic-text alignment module. The Transformer block utilizes a cross-attention mechanism to align topic features with topic image features, ensuring that the topic feature encoder can identify the corresponding topic content from the reference image based on the topic text. Specifically, to improve the expressive power of topic features, feature-text alignment modules are introduced at different scales in UNet. Fine-grained feature-text alignment is used to extract multi-scale topic features, i.e., the output of the feature-text alignment module at each scale. All of these will be used as the final extracted subject image features. It is a component of.

[0031] (2) Theme layout generation module.

[0032] Traditional image subject layout methods have limitations due to their reliance on manually drawing complex masks or text prompts. To overcome this deficiency and generate correct and reasonable image layouts, this invention introduces a subject layout generation module into the single-subject feature extractor to automatically generate a reasonable image layout based on the subject and reference image.

[0033] Since the aforementioned topic feature encoder is trained based on the reconstruction loss of the topic mask image, the trained topic feature encoder possesses the ability to reconstruct the topic mask image and mask other topics and the background. Therefore, this invention utilizes the reconstruction and masking capabilities of a single-topic encoder to obtain the regional location of the topic in the reference image, and generates a topic layout to guide the generation of multi-topic images based on the layout of the reference image.

[0034] Specifically, the layout generation algorithm of the theme layout generation module can be described as follows: Decoding Topic Image Features: Among the multi-scale topic features extracted by the topic feature encoder, the largest-scale topic feature is the vector representation of the topic mask image in the VAE encoding space. Therefore, the topic layout generation module uses the VAE decoder to decode the topic image features back into pixel space, obtaining a mask image containing only the topic.

[0035] Generating the minimum bounding matrix: In pixel space, since only the pixels in the subject area of ​​the mask image are non-zero, the subject position can be determined by judging the regions with non-zero pixels. Specifically, a depth-first traversal algorithm is used to obtain the largest 8-connected graph with non-zero pixel values ​​in the mask image, and the minimum bounding matrix of the 8-connected graph is generated.

[0036] Overlaying to generate subject position layout: After generating the minimum bounding matrix, map it onto the reference image. In the coordinate system, the specific location of the subject in the reference image is determined.

[0037] Multi-topic position fusion: After generating the minimum bounding matrix of a single topic and determining its position in the reference image, the topic layout generation module overlays and fuses the minimum bounding matrices of multiple topics. By fusing the layouts of multiple topics, the relative positions of each topic are ensured to be reasonable and consistent with the reference image.

[0038] Final Position Layout Generation: After the fusion of multiple subject positions, the subject layout generation module outputs the final subject position layout. This layout will serve as a key reference to guide the generation of multi-subject images, ensuring that the generated images are reasonable and meet expectations in terms of subject position and layout.

[0039] In summary, the theme layout generation module automatically generates a reasonable image layout through steps such as decoding theme image features, generating a minimum bounding matrix, superimposing to generate theme position layouts, and fusing multiple theme positions. This overcomes the shortcomings of traditional methods that require manual provision of complex mask layouts and reliance on text-based layout generation, achieving efficient and accurate multi-theme image layout generation and providing reasonable guidance information for subsequent multi-theme image generation.

[0040] S2, construct a region feature injection module for extracting layout information, including a region feature masking submodule and a region feature fusion submodule; the region feature masking submodule performs explicit region division of latent space features according to the theme layout information; the region feature fusion submodule fuses the image features of each theme with the corresponding region features.

[0041] In this embodiment of the invention, the module includes two sub-modules: a region feature mask and a region feature fusion module. The region feature mask module explicitly divides the latent spatial features in the denoising network into regions based on the spatial layout information of each theme, ensuring that different theme features are independently injected into their corresponding generation regions. The region feature fusion module is responsible for efficiently fusing the image features of each theme with its corresponding region features. During the fusion process, a cross-attention mechanism is used to effectively improve the semantic correlation between theme features and spatial regions, prevent feature interference between different themes, and achieve isolation and targeted expression of multi-theme features. Through the synergistic effect of the above sub-modules, the region feature injection module can significantly improve the recognizability and spatial distribution rationality of each theme in the generated image, ensuring high fidelity and clear semantic boundaries in multi-theme image generation tasks.

[0042] Reference Figure 3 The following is a detailed description of each submodule.

[0043] (1) Regional feature masking submodule. In this embodiment of the invention, the main function of the region feature masking submodule is to explicitly divide the latent space features into regions based on the input topic layout information. This module ensures that each topic feature acts on its dedicated generation region through spatial constraints, thereby effectively avoiding semantic conflicts between multiple topic features and providing corresponding region feature inputs for subsequent cross-attention.

[0044] Specifically, the region feature mask submodule can be described as follows: Region feature masking stage receiving topic layout Features of latent space As input. Theme layout It is a spatial prior used to define the generated region for each topic in a multi-topic image. It is represented as a binary mask, where a value of 1 in each mask matrix corresponds to the generated region for that specific topic, and a value of 0 represents a non-generated region. Latent spatial features It is a global feature representation extracted from the latent space in the denoising network, containing the initial semantic information of the image.

[0045] Next, based on the input topic layout information, latent space features... The latent space features are divided into several regional features. Specifically, using a binary mask matrix of the topic layout, each pixel of the latent space feature is multiplied pixel-by-pixel with the mask value to obtain multiple local feature maps corresponding to the generation region of each topic. For a two-topic task, the latent space features... The resolution is mask matrix and Let the generated regions of Topic 1 and Topic 2 be represented respectively. The calculation of local features is shown in Equations (1) and (2):

[0046] in, This indicates pixel-by-pixel multiplication. and These are the local features of the divided theme 1 and theme 2.

[0047] In one embodiment of the invention, the region masking module explicitly partitions the latent space features using topic layout information, thereby making the generation regions of each topic independent. This spatial segmentation strategy avoids confusion between multi-topic features in the early stages of feature generation, allowing each topic feature to focus on its corresponding generation region, thus significantly reducing the probability of semantic conflicts. The region masking module is highly flexible in design; both the number of topics and the layout can be adjusted according to task requirements. This modular design enables it to adapt to various multi-topic generation tasks and seamlessly integrate with other functional modules such as the cross-attention module.

[0048] (2) Regional feature fusion submodule.

[0049] In this embodiment of the invention, the core function of the region feature fusion submodule is to achieve efficient feature fusion between region features and subject features. For multi-subject image generation tasks, this module enhances the semantic expressive power of region features through a cross-attention mechanism, while effectively avoiding interference between different subject features, achieving feature isolation, and thus ensuring the high fidelity of the generated image and the independence of subject features.

[0050] The input to the region feature fusion submodule includes region features. With thematic features ,in Regional characteristics Generated by the region feature mask submodule described in (2.1), representing the local features of each topic in a specific region; topic features The topic features are extracted by the topic feature encoder described in (1). Thematic image features, containing semantic information about the theme, are used to guide the generation and optimization of regional features.

[0051] To achieve interaction between region features and latent space features, the region feature fusion adopts the classic cross-attention mechanism, where region features are used as query vectors and topic features are used as key-value vectors. The specific module calculation process includes the following steps: Input and Feature Flattening: Since the topic features and region features have different dimensions, both input features are flattened into a two-dimensional matrix form before entering the cross-attention calculation, as shown in Equations (3) and (4):

[0052] The flattening operation transforms the features from a three-dimensional form into a two-dimensional matrix form, which facilitates subsequent attention calculations.

[0053] Cross-attention computation: In the cross-attention module, region features are used as the query vector, and topic features are used as the key and value vectors. Information exchange between the two is achieved through a cross-attention mechanism. To ensure consistency in the feature dimensions of the query, key, and value during cross-attention computation, region features... and thematic features First, three different linear transformations are applied to map to the same feature dimension. As shown in equations (5), (6) and (7):

[0054] in, is a linear transformation matrix. The cross-attention layer measures the correlation between regional features and topic features based on the dot product similarity between the query and the key vector, and performs a weighted summation of the value vectors based on the correlation, aggregating the contextual information of the relevant topics in the current region into the regional features, as shown in Equation (8):

[0055] in, The merged regional features integrate interactive information between regional features and thematic features.

[0056] Residual Connections and Output: To preserve the information of the original region features, residual connections are used. First, a linear layer is used to... The dimensions are restored to their original size, and then the original region features are flattened. Features processed by cross-attention mechanism Perform a weighted summation, as shown in equation (9):

[0057] in, The output features of the region feature fusion module integrate the semantic information of the original region features and topic features. This is the fusion coefficient, used to control the degree to which original regional features are preserved.

[0058] S3 employs a two-stage training approach: the first stage trains the single-topic feature extractor, and the second stage jointly trains the denoising UNet and the single-topic feature extractor.

[0059] The location-guided multi-topic defect image generation method proposed in this invention employs a two-stage training approach. The first stage trains a single-topic feature extractor based on the image, topic text, and topic mask, enabling the pre-trained single-topic feature extractor to extract topic features and generate topic layouts based on the topic text. The second stage combines the denoising UNet with the single-topic feature extractor for joint training, allowing the trained overall model to generate reasonable and clearly defined images based on topic features and topic layouts.

[0060] The following provides a detailed explanation of each training stage.

[0061] (1) First stage.

[0062] Since the single-topic feature extractor introduces a topic mask during training, in order for the model to effectively distinguish the topic content from the background region and extract relevant feature information, the single-topic feature extractor pre-training uses supervised learning to guide the encoder to extract the corresponding features. Mean Squared Error (MSE) loss is calculated using the real features and the extracted features, and the encoding model is trained through backpropagation. The loss function is shown in Equation (10):

[0063] in, The features of the real subject image after subject masking. Multi-scale topic image features reconstructed by the topic feature encoder. For the largest scale feature, its resolution is... Maintain consistency. Furthermore, real-world subject image features. With reconstructing subject image features The calculation methods are shown in equations (11) and (12) respectively:

[0064] in, For image encoders, The image from which features are to be extracted. The content mask is a subject line that takes the value 0 or 1. This is the UNet network used to extract topic features.

[0065] (2) Second stage.

[0066] In order to generate multi-topic images with corresponding layouts based on the extracted topic images and topic layouts, the pre-trained single-topic feature extractor and the multi-scale region feature injection module are jointly trained, and the loss function is shown in Equation (13):

[0067] in, It is real noise sampled from a standard normal distribution. The noise at the current moment is predicted by the denoising model. The embedding vector for the text prompt. In time step The noisy latent layer vector, To inject the set of topic features into the denoising network, The layout is based on the theme. It can be expressed as equation (14):

[0068] Among them, the Thematic characteristics It is a set of multi-scale features, that is:

[0069] This concludes the explanation of the second phase of training.

[0070] S4, based on the extracted thematic features and layout information, guides and denoises the latent space variables to generate high-quality multi-theme images.

[0071] Ultimately, in this embodiment of the invention, a trained single-topic feature extractor and a region feature injection module are used to extract topic features and layout information. By guiding and denoising the latent space variables, high-quality synthesis of multi-topic images is achieved, resulting in a generated image.

[0072] Furthermore, in this embodiment of the invention, the experimental environment and evaluation indicators are as follows: A. Experimental environment: On the hardware side, an NVIDIA A100 was used for model training and validation. On the software side, the system used Ubuntu 20.04, Python version 3.8, PyTorch version 2.4.1, and CUDA version 11.8.

[0073] Stable-diffusion-v1-4 was used as the pre-trained diffusion model, supplemented by a pre-trained Clip-vit-large-patch14 as the text encoder. For training, the dimensions of all images were resized to [value missing]. The pixels are then input into the subject feature extractor. DDIM is used as the sampler to sample the latent vectors, with a stride of 50 and a guidance scale of 7.5.

[0074] B Evaluation Indicators: Use DINO-M, CLIP-M, CLIP-T, and aesthetic deviation levels.

[0075] DINO-M: For topic similarity metrics in multi-topic images, since both the input and output images contain multiple topics, directly comparing the similarity between the input and output images is inevitably inaccurate. Therefore, for each generated image, a masked version of its topic is obtained, and then the masked version of the input image is compared with the masked version of the generated image to obtain a topic similarity score. Its calculation method can be expressed as:

[0076] in, and The generated image is the first The subject mask portion and the reference image's first Image features of a subject mask portion in DINO space.

[0077] CLIP-M: Similarly, CLIP-M is used to measure the similarity of topics in multi-topic images. It calculates the cosine similarity of the masked versions of each topic image in the CLIP space. Its calculation method can be represented as:

[0078] in, and The generated image is the first The subject mask portion and the reference image's first Image features of a subject mask portion in CLIP space.

[0079] CLIP-T: This method measures the alignment between a prompt and a synthesized image by calculating the cosine similarity between the feature vectors of the generated image and the prompt in the CLIP feature space. The calculation can be represented as follows:

[0080] in, To generate image features in CLIP space, These are the textual features of the prompt words in the CLIP space.

[0081] Degree of Aesthetic Deviation (DAS): The specific calculation method is as follows:

[0082] in, This represents the aesthetic score of the synthesized image. This represents the aesthetic score of the real image. The aesthetic score is obtained by scoring the generated image using the pre-trained aesthetic evaluation model LAION-Aesthetics Predictor (LAP), which mainly considers visual elements such as composition, color, and sharpness, with a score range of 0 to 10.

[0083] Obviously, those skilled in the art should understand that the steps of the location-guided multi-topic defect image generation method or the modules of the location-guided multi-topic image generation system described in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

[0084] To implement the methods of the above embodiments, the present invention also provides an electronic device, which includes a memory and a processor; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the various steps of the methods described above.

[0085] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0086] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0087] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0088] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A location-guided method for generating multi-topic defect images, characterized in that, Includes the following steps: A single-topic feature extractor for extracting topic features is constructed, including a topic feature encoder and a topic layout generation module; the topic feature encoder is used to jointly encode the input reference image and topic text to extract highly expressive image features of each topic; the topic layout generation module automatically generates spatial layout information of each topic based on the extracted topic features. A region feature injection module for extracting layout information is constructed, including a region feature masking submodule and a region feature fusion submodule; the region feature masking submodule performs explicit region division of latent space features according to the theme layout information; the region feature fusion submodule fuses the image features of each theme with the corresponding region features; A two-stage training approach is adopted: the first stage trains the single-topic feature extractor, and the second stage jointly trains the denoising UNet and the single-topic feature extractor. Based on the extracted thematic features and layout information, latent space variables are guided and denoised to generate high-quality multi-theme images.

2. The location-guided multi-topic defect image generation method according to claim 1, characterized in that, The topic feature encoder is specifically used for: During the coding phase, input the topic text. Compared with reference image The topic text is encoded using the CLIP text encoder to obtain the text embedding. The reference image is encoded using a VAE image encoder to obtain the image embedding. ; Spatial feature enhancement is achieved by extracting topic features from the spatial self-attention module within the UNet structure. This spatial self-attention module specifically comprises two residual modules, two Transformer blocks, and a downsampling unit. The first spatial self-attention module receives the reference image embedding. As input, where C is the number of feature channels, H is the height, and W is the width, the subsequent spatial self-attention module receives the image features enhanced by the previous spatial self-attention module as input; Topic feature extraction is achieved through the topic-text alignment module in UNet. The topic-text alignment module specifically includes two residual modules, two Transformer blocks, and an upsampling unit. The topic-text alignment module receives spatial self-attention image features from the spatial attention module at the same level, topic text features encoded by the CLIP text encoder, and topic image features extracted by the deep topic-text alignment module. The Transformer block uses a cross-attention mechanism to align the topic features with the topic image features.

3. The location-guided multi-topic defect image generation method according to claim 1, characterized in that, The theme layout generation module is specifically used for: The VAE decoder is used to decode the features of the subject image back into pixel space to obtain a mask image containing only the subject. The maximum 8-connected graph with non-zero pixel values ​​in the mask image is obtained by using a depth-first traversal algorithm, and the minimum bounding rectangle of the 8-connected graph is generated. Map the smallest bounding rectangle onto the coordinate system of the reference image to determine the specific location of the subject in the reference image; Overlay and merge the minimum bounding rectangles of multiple themes; The final topic location layout is output as a spatial prior for multi-topic image generation.

4. The location-guided multi-topic defect image generation method according to claim 3, characterized in that, The region feature mask submodule is specifically used for: Based on the input topic layout information, the latent space features are explicitly divided into regions. Using the binary mask matrix of the topic layout, each pixel of the latent space feature is multiplied with the mask value pixel by pixel to obtain multiple local feature maps.

5. The location-guided multi-topic defect image generation method according to claim 4, characterized in that, The region feature fusion submodule is specifically used for: Efficient fusion between regional features and topic features is achieved through a cross-attention mechanism. Regional features are used as query vectors, and topic features are used as key vectors. The fused regional features are output using a residual connection method.

6. The location-guided multi-topic defect image generation method according to claim 5, characterized in that, The two-stage training method specifically includes: The first stage involves pre-training a single-topic feature extractor, using supervised learning to guide the encoder to extract features, calculating the mean squared error loss using the real features and the extracted features, and training the encoding model through backpropagation. The second stage involves joint training of the entire model, which combines the pre-trained single-topic feature extractor with the multi-scale region feature injection module, enabling the model to generate multi-topic images with corresponding layouts based on the extracted topic images and topic layouts.

7. An electronic device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-6.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images