Pet image generation method based on multi-label diffusion model

Abstract

The invention discloses a pet image generation method based on a multi-label diffusion model, and relates to the technical field of artificial intelligence image generation, and the method comprises the steps: carrying out the feature extraction of a to-be-processed pet image, and obtaining a pet image feature map; performing down-sampling processing on the feature map input down-sampling layer; inputting the picture information subjected to the down-sampling processing into a middle layer; inputting the picture information processed by the middle layer into an upper sampling layer; wherein the lower sampling layer and the upper sampling layer comprise a multi-label self-adaptive module, and the multi-label self-adaptive module comprises a first convolution module, a second convolution module and a third convolution module; and the multi-label vector joint superposition module, the second convolution module and the attention module are used for automatically capturing the incidence relation between the multi-label vectors and the intermediate features. A multi-label coding module and a multi-label adaptive module are added and used for guiding a diffusion model to generate a high-quality pet image with multi-label features.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence image generation, and in particular to a pet image generation method based on a multi-label diffusion model. Background Art

[0002] With the rapid development of artificial intelligence technology, image generation technology has been widely used in many fields. Especially in the field of pet image generation, traditional generation methods often fail to meet users' high requirements for image quality and diversity. Existing image generation technology mainly relies on a single label for image generation, which cannot fully consider the diversity and complexity of pet images.

[0003] As an emerging generative model, the diffusion model has demonstrated powerful capabilities in image generation tasks. However, existing diffusion models still have certain limitations when processing multi-label information and are difficult to generate high-quality images that meet multiple feature requirements. Summary of the invention

[0004] The present invention provides a pet image generation method based on a multi-label diffusion model to generate high-quality images that meet multiple feature requirements.

[0005] According to one aspect of the present disclosure, a pet image generation method based on a multi-label diffusion model is provided, the method comprising: Perform feature extraction on the pet image to be processed to obtain a pet image feature map; Performing downsampling processing on the feature map input downsampling layer; Input the downsampled image information into the middle layer; Input the image information processed by the intermediate layer to the upsampling layer; Among them, the downsampling layer and the upsampling layer include a multi-label adaptive module, which includes: performing a convolution operation on the feature map through a first convolution module to extract local features to obtain intermediate features; jointly superimposing the intermediate features with the multi-label vector, and integrating the information of the pet's multi-label features into the intermediate features; inputting the intermediate features that integrate the pet's multi-label features into the second convolution module; and inputting the output of the second convolution module into the attention module to automatically capture the correlation between the multi-label vector and the intermediate features.

[0006] In a possible implementation, the method further includes: encoding the multi-label vector by a multi-label encoding module, including: Encode the diffusion step number t, pet type label c1, pet color label c2, and pet body shape label c3 to generate a 4×64 dimensional label vector. The mathematical expression of the encoding layer is shown in formula (1): (1) Among them, E represents the vector mapping function, which maps the vector (t, c1, c2, c3) to the vector dimension required by the model, w is the linear transformation matrix, b is the bias coefficient, and σ is the Swish activation function.

[0007] In a possible implementation, the first convolution module includes: a normalization layer, a first convolution layer, an activation layer, and a second convolution layer.

[0008] In a possible implementation, the second convolution module has the same structure as the first convolution module.

[0009] In one possible implementation, the attention module is a self-attention layer.

[0010] In a possible implementation, the multi-label vector includes the number of diffusion steps, the pet type label, the pet color label, and the pet body shape label.

[0011] Compared with the prior art, the present invention has the following beneficial effects: Designing a diffusion model network structure that can use multi-label information for image generation has become an important direction of current research. The diffusion model usually adopts the U-Net network structure, which consists of a downsampling layer, an intermediate layer, and an upsampling layer. The downsampling layer extracts feature vectors while reducing the resolution of the feature map, the intermediate layer is mainly responsible for extracting features, and the upsampling block extracts features while increasing the feature resolution.

[0012] This paper proposes a pet image generation method based on a multi-label diffusion model. This method improves on the diffusion model and designs a multi-label diffusion network structure. It can use multiple labels (such as pet type, color, and body shape) to guide the generated images and generate more diverse high-quality pet images.

[0013] The multi-label diffusion model proposed in the present invention designs an improved U-Net network structure. Different from the traditional diffusion model network structure U-Net, the present invention adds a multi-label encoding module and a multi-label adaptive module to guide the diffusion model to generate high-quality pet images with multi-label features. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic diagram showing a pet image generation method based on a multi-label diffusion model according to an embodiment of the present disclosure.

[0015] Figure 2 A block diagram of a multi-label adaptive module according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0016] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0017] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0018] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0019] According to one aspect of the present disclosure, a pet image generation method based on a multi-label diffusion model is provided, the method comprising: Perform feature extraction on the pet image to be processed to obtain a pet image feature map; Performing downsampling processing on the feature map input downsampling layer; Input the downsampled image information into the middle layer; Input the image information processed by the intermediate layer to the upsampling layer; Among them, the downsampling layer and the upsampling layer include a multi-label adaptive module, which includes: performing a convolution operation on the feature map through a first convolution module to extract local features to obtain intermediate features; jointly superimposing the intermediate features with the multi-label vector, and integrating the information of the pet's multi-label features into the intermediate features; inputting the intermediate features that integrate the pet's multi-label features into the second convolution module; and inputting the output of the second convolution module into the attention module to automatically capture the correlation between the multi-label vector and the intermediate features.

[0020] In a possible implementation, the method further includes: encoding the multi-label vector by a multi-label encoding module, including: Encode the diffusion step number t, pet type label c1, pet color label c2, and pet body shape label c3 to generate a 4×64 dimensional label vector. The mathematical expression of the encoding layer is shown in formula (1): (1) Among them, E represents the vector mapping function, which maps the vector (t, c1, c2, c3) to the vector dimension required by the model, w is the linear transformation matrix, b is the bias coefficient, and σ is the Swish activation function.

[0021] In a possible implementation, the first convolution module includes: a normalization layer, a first convolution layer, an activation layer, and a second convolution layer.

[0022] In a possible implementation, the second convolution module has the same structure as the first convolution module.

[0023] In one possible implementation, the attention module is a self-attention layer.

[0024] In a possible implementation, the multi-label vector includes the number of diffusion steps, the pet type label, the pet color label, and the pet body shape label.

[0025] like Figure 1 As shown in the figure, the multi-label encoding module (multi-label encoding layer) is added at the initial position of the network to encode the diffusion step number t, pet type label c1, pet color label c2, and pet body shape label c3 to generate a 4×64 dimensional label vector. The mathematical expression of its encoding layer is as follows: (1) Where E represents the vector mapping function, which maps (t, c1, c2, c3) to the vector dimension required by the model, w is the linear transformation matrix, b is the bias coefficient, is the Swish activation function.

[0026] Multi-label adaptive module (multi-label encoding layer): added in each upsampling layer and downsampling layer, the image feature map x and the multi-label encoding vector (t, c1, c2, c3) are fused, so that the multi-label guided diffusion model can be used to generate pet images with multi-label features. The network structure of this module is as follows Figure 2 As shown: The input feature x_i first passes through a convolution module consisting of a normalization layer GN, a convolution layer Conv3×3,1, an activation layer ReLU, and a convolution layer Conv1×1,1. The function of this convolution module is to perform a convolution operation on the feature input feature, extract its local features, and enhance its expressiveness. The intermediate features processed by the convolution module are jointly superimposed with the multi-label vector (t,c1,c2, c3), that is, the information of the pet's multi-label features is integrated into the intermediate features. The fused intermediate features are then processed by a convolution module and input into a self-attention layer to automatically capture the correlation between the multi-label vector and the intermediate features, thereby improving the adaptive ability of the fused feature a_i.

[0027] Among them, Xi is the feature vector extracted by the network before the multi-label adaptive module, and ai is the feature vector obtained after the multi-label adaptive module.

[0028] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A pet image generation method based on a multi-label diffusion model, characterized in that: The method comprises: Perform feature extraction on the pet image to be processed to obtain a pet image feature map; Performing downsampling processing on the feature map input downsampling layer; Input the downsampled image information into the middle layer; Input the image information processed by the intermediate layer to the upsampling layer; Among them, the downsampling layer and the upsampling layer include a multi-label adaptive module, which includes: performing a convolution operation on the feature map through a first convolution module to extract local features to obtain intermediate features; jointly superimposing the intermediate features with the multi-label vector, and integrating the information of the pet's multi-label features into the intermediate features; inputting the intermediate features that integrate the pet's multi-label features into the second convolution module; and inputting the output of the second convolution module into the attention module to automatically capture the correlation between the multi-label vector and the intermediate features.

2. The pet image generation method based on a multi-label diffusion model according to claim 1, characterized in that: The method further includes: encoding the multi-label vector by a multi-label encoding module, including: Encode the diffusion step number t, pet type label c1, pet color label c2, and pet body shape label c3 to generate a 4×64 dimensional label vector. The mathematical expression of the encoding layer is shown in formula (1): （1） Among them, E represents the vector mapping function, which maps the vector (t, c1, c2, c3) to the vector dimension required by the model, w is the linear transformation matrix, b is the bias coefficient, and σ is the Swish activation function.

3. The pet image generation method based on a multi-label diffusion model according to claim 1, characterized in that: The first convolution module includes: a normalization layer, a first convolution layer, an activation layer, and a second convolution layer.

4. The pet image generation method based on a multi-label diffusion model according to claim 1, characterized in that: The structure of the second convolution module is the same as that of the first convolution module.

5. The pet image generation method based on a multi-label diffusion model according to claim 1, characterized in that: The attention module is a self-attention layer.

6. The pet image generation method based on a multi-label diffusion model according to claim 1, characterized in that: The multi-label vector includes the number of diffusion steps, pet type label, pet color label, and pet body shape label.

Images