Image generation model training method, image generation method and electronic equipment

Abstract

The embodiment of the invention provides an image generation model training method, an image generation method and electronic equipment, and relates to the technical field of computer vision, and the method comprises the steps: obtaining a plurality of sample pairs; each sample pair comprises two sample images which are the same in size and both comprise objects of a preset type; the two sample images are respectively provided with a labeling frame for representing an image area occupied by the object, and the positions and the sizes of the labeling frames in the two sample images are the same; inputting a sample image in each sample pair and a sample description text used for describing a second sample image in the sample pair into an image generation model of the initial structure to obtain a prediction image; and on the basis of the difference between a predicted image obtained by a diffusion network in the image generation model of the initial structure and a second sample image in the sample pair when each sample pair is processed, adjusting model parameters until a preset convergence condition is reached, and obtaining a trained image generation model. Therefore, the efficiency of acquiring the sample image can be improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image generation model training method, an image generation method, and an electronic device. Background Technology

[0002] With the development of computer vision technology, its application scenarios are becoming increasingly widespread, such as target (e.g., human face or animal) detection. In target detection, deep learning models can be used. To ensure the model's generalization ability, a large number of sample images are often needed to train the model; therefore, the training sample set needs to be expanded. Related technologies can generate images using pre-trained sample generation models, such as GANs (Generative Adversarial Networks) or diffusion models. Subsequently, the objects to be detected (the objects to be detected) in the generated images need to be manually labeled, which consumes significant manual labor and is inefficient, resulting in low efficiency in acquiring sample images. Summary of the Invention

[0003] The purpose of this application is to provide an image generation model training method, an image generation method, and an electronic device to reduce labor costs and improve the efficiency of acquiring sample images. The specific technical solution is as follows:

[0004] A first aspect of this application provides a method for training an image generation model, the method comprising:

[0005] Multiple sample pairs are obtained; wherein each sample pair includes two sample images of the same size, both containing objects of a preset type; both sample images have bounding boxes representing the image area occupied by the object, and the bounding boxes of the two sample images are in the same position and size in their respective images.

[0006] Input the first and second sample images of each sample pair, along with the sample description text used to describe the second sample image in the sample pair, into the image generation model of the initial structure to obtain the predicted image output by the image generation model of the initial structure.

[0007] Based on the difference between the predicted image obtained by the diffusion network in the initial image generation model and the second sample image in the sample pair when processing each sample pair, the model parameters of the initial image generation model are adjusted until the preset convergence condition is reached, and the trained image generation model is obtained.

[0008] Optionally, the step of inputting the first and second sample images of each sample pair, along with sample description text describing the second sample image in the sample pair, into the image generation model of the initial structure to obtain the predicted image output by the image generation model of the initial structure includes:

[0009] The following steps are performed using an image generation model based on the initial structure:

[0010] Image features are extracted from the first sample image in each sample pair to obtain sample image features;

[0011] Based on the position of the bounding box in the first sample image, the first sample image is binarized to obtain a sample mask image;

[0012] Text features are extracted from the sample description text used to describe the second sample image in the sample pair to obtain the sample text features;

[0013] The second sample image in the sample pair is forward diffused through the diffusion network to obtain the processing result of the first sample.

[0014] The first sample processing result is back-diffused using the diffusion network, the sample mask image, the sample image features, and the sample text features to obtain a predicted image.

[0015] Optionally, the second sample image in the sample pair is forward diffused through the diffusion network to obtain the first sample processing result, including: generating a specified number of preset noises through the diffusion network and adding them sequentially to the second sample image in the sample pair to obtain the first sample processing result;

[0016] The process involves using the diffusion network to perform back-diffusion processing on the first sample processing result, utilizing the sample mask image, sample image features, and sample text features, to obtain a predicted image. This includes: fusing the current sample fusion result with the sample mask image using the diffusion network to obtain a current first sample fusion result; wherein the sample fusion result used in the first fusion to obtain the first sample fusion result is the first sample processing result; processing the current first sample fusion result using the sample image features and sample text features based on a cross-attention mechanism using the diffusion network to obtain a current second sample processing result; denoising the current sample fusion result using the current second sample processing result using the diffusion network to obtain a new sample fusion result, and then returning to the step of fusing the current sample fusion result with the sample mask image using the diffusion network to obtain the current first sample fusion result, until the number of denoising operations reaches the specified number, resulting in a predicted image.

[0017] The step of adjusting the model parameters of the initial image generation model based on the difference between the predicted image obtained by the diffusion network in the initial image generation model and the second sample image in the sample pair when processing each sample pair, until a preset convergence condition is met, to obtain a trained image generation model, includes: for each second sample processing result, calculating the sub-loss value corresponding to the second sample processing result based on the difference between the second sample processing result and the corresponding preset noise; wherein, the i-th second sample processing result obtained in the denoising order corresponds to the (n+1)-i-th added preset noise, and n represents a specified number; calculating the loss value based on the sub-loss value corresponding to each second sample processing result; and adjusting the model parameters of the initial image generation model according to the calculated loss value, until a preset convergence condition is met, to obtain a trained image generation model.

[0018] Optionally, the image generation model of the initial structure further includes: a text feature extraction layer and an image feature extraction layer, and the diffusion network includes: an encoding layer, a decoding layer, a first multi-head attention layer and a second multi-head attention layer;

[0019] For each sample pair, image features are extracted from the first sample image to obtain sample image features, including:

[0020] The first sample image in each sample pair is input into the image feature extraction layer to obtain the sample image features;

[0021] The text feature extraction of the sample description text used to describe the second sample image in the sample pair is performed to obtain sample text features, including: inputting the sample description text used to describe the second sample image in the sample pair into the text feature extraction layer to obtain sample text features;

[0022] The diffusion network, based on a cross-attention mechanism, processes the current first sample fusion result using the sample image features and the sample text features to obtain the current second sample processing result. This process includes: encoding the current first sample fusion result using the encoding layer to obtain a sample encoding result; processing the obtained sample encoding result and the sample image features using the first multi-head attention layer to obtain a first sample attention result; processing the obtained sample encoding result and the sample text features using the second multi-head attention layer to obtain a second sample attention result; fusing the first sample attention result and the second sample attention result to obtain a second sample fusion result; and decoding the obtained second sample fusion result using the decoding layer to obtain the current second sample processing result.

[0023] Optionally, the text feature extraction layer is the part of the pre-trained image-text pre-training model used to extract text features, and the parameters of the text feature extraction layer remain unchanged during the training of the image generation model with the initial structure.

[0024] Optionally, the diffusion network is implemented based on Unet, and a first multi-head attention layer and a second multi-head attention layer are set between each encoding layer and its corresponding decoding layer; the query of the first and second multi-head attention layers corresponding to each encoding layer is obtained based on the output data of that encoding layer; the data obtained by performing different linear transformations on the sample image features are respectively the key and value of the first multi-head attention layer; the data obtained by performing different linear transformations on the sample text features are respectively the key and value of the second multi-head attention layer;

[0025] And / or,

[0026] The method further includes: using the image feature extraction layer in the trained image generation model to extract features from a first image to be detected containing the preset type of object, thereby obtaining first image features to be detected; wherein the first image features to be detected are used to perform image detection on the first image to be detected.

[0027] Optionally, a sample pair is obtained based on the following steps: selecting any two images from pre-acquired images containing objects of the preset type and having the object's bounding box; transforming the two selected images so that the transformed images have the same size, and the bounding boxes of the two images have the same position and size in their respective images, thus obtaining a sample pair.

[0028] A second aspect of this application also provides an image generation method, the method comprising:

[0029] Obtain a reference image containing objects of a preset type, and a bounding box representing the image area occupied by the objects of the preset type;

[0030] The reference image, preset descriptive text, and preset noise image are input into a pre-trained image generation model to obtain a usable image that conforms to the preset descriptive text and contains an object of the preset type; wherein, the image region occupied by the object of the preset type in the usable image is the same in position and size as the image region occupied by the object of the preset type in the reference image; the preset noise image is the same size as the reference image; the image generation model is trained based on any of the methods described in the first aspect above.

[0031] Optionally, the reference image, preset descriptive text, and preset noisy image are input into a pre-trained image generation model to obtain a usable image that conforms to the preset descriptive text and contains an object of the preset type, including:

[0032] The following steps are performed using a pre-trained image generation model: image feature extraction is performed on the reference image to obtain the image features to be used; the reference image is binarized according to the position of the bounding boxes in the reference image to obtain the mask image to be used; text feature extraction is performed on the preset descriptive text to obtain the text features to be used; and the preset noisy image is back-diffused using the mask image to be used, the image features to be used, and the text features to be used through a diffusion network to obtain the image to be used that conforms to the preset descriptive text and contains the object of the preset type.

[0033] Optionally, the step of using a diffusion network to perform back-diffusion processing on the preset noisy image using the mask image to be used, the features of the image to be used, and the features of the text to be used to obtain a usable image that conforms to the preset descriptive text and contains an object of the preset type includes:

[0034] The current fusion result to be utilized is fused with the mask image to be utilized through a diffusion network to obtain the current first fusion result to be utilized; wherein, the fusion result to be utilized used when the first fusion is performed to obtain the first fusion result to be utilized is the preset noise image;

[0035] The diffusion network, based on the cross-attention mechanism, processes the current first fusion result to be utilized using the image features and text features to be utilized, thereby obtaining the current processing result to be utilized.

[0036] The diffusion network is used to denoise the current first fusion result to be utilized using the current processing result to obtain a new fusion result to be utilized. Then, the process returns to the step of fusing the current fusion result to be utilized with the mask image to be utilized using the diffusion network to obtain the current first fusion result to be utilized. This process continues until the number of denoising operations reaches a specified number, resulting in a fusion image to be utilized that conforms to the preset description text and contains objects of the preset type.

[0037] Optionally, the image generation model further includes a text feature extraction layer and an image feature extraction layer, and the diffusion network includes an encoding layer, a decoding layer, a first multi-head attention layer, and a second multi-head attention layer;

[0038] Image feature extraction is performed on the reference image to obtain the image features to be used, including:

[0039] The reference image is input into the image feature extraction layer to obtain the image features to be used;

[0040] Text feature extraction is performed on the preset descriptive text to obtain the text features to be utilized, including:

[0041] Input the preset descriptive text into the text feature extraction layer to obtain the text features to be utilized;

[0042] The diffusion network, based on a cross-attention mechanism, processes the current first fusion result to be utilized using the image features and text features to be utilized, to obtain the current processing result to be utilized. This process includes: encoding the current first fusion result to be utilized using the encoding layer to obtain an encoded result to be utilized; processing the obtained encoded result and the image features to be utilized using the first multi-head attention layer to obtain a first attention result to be utilized; processing the obtained encoded result and the text features to be utilized using the second multi-head attention layer to obtain a second attention result to be utilized; fusing the first attention result and the second attention result to obtain a second fusion result to be utilized; and decoding the second fusion result to obtain the current processing result to be utilized.

[0043] Optionally, the diffusion network is implemented based on Unet, and a first multi-head attention layer and a second multi-head attention layer are set between each encoding layer and its corresponding decoding layer; the query of the first multi-head attention layer and the second multi-head attention layer corresponding to each encoding layer is obtained based on the output data of that encoding layer, and the data obtained by performing different linear transformations on the image features to be utilized are respectively the key and value of the first multi-head attention layer; the data obtained by performing different linear transformations on the text features to be utilized are respectively the key and value of the second multi-head attention layer;

[0044] And / or,

[0045] The method further includes: using the image feature extraction layer to extract features from a second image to be detected containing an object of the preset type, thereby obtaining features of the second image to be detected; and inputting the features of the second image to be detected into a classification layer to obtain the image detection result of the second image to be detected.

[0046] A third aspect of this application also provides an image generation model training apparatus, the apparatus comprising:

[0047] The sample pair acquisition module is used to acquire multiple sample pairs; wherein each sample pair includes two sample images of the same size, both containing objects of a preset type; both sample images have bounding boxes representing the image area occupied by the objects, and the bounding boxes of the two sample images are in the same position and size in their respective images.

[0048] The prediction image determination module is used to input the first sample image and the second sample image in each sample pair, as well as the sample description text used to describe the second sample image in the sample pair, into the image generation model of the initial structure to obtain the prediction image output by the image generation model of the initial structure.

[0049] The model parameter adjustment module is used to adjust the model parameters of the initial image generation model based on the difference between the predicted image obtained by the diffusion network in the initial image generation model and the second sample image in the sample pair when processing each sample pair, until the preset convergence condition is reached, and the trained image generation model is obtained.

[0050] Optionally, the predicted image determination module is specifically used to perform the following steps using the image generation model with the initial structure:

[0051] For each sample pair, image features are extracted from the first sample image to obtain sample image features; the first sample image is binarized according to the position of the bounding box in the first sample image to obtain a sample mask image; text features are extracted from the sample description text used to describe the second sample image in the sample pair to obtain sample text features; the second sample image in the sample pair is forward diffused through the diffusion network to obtain the first sample processing result; the first sample processing result is backdiffused through the diffusion network using the sample mask image, the sample image features, and the sample text features to obtain the predicted image.

[0052] Optionally, the second sample image in the sample pair is forward diffused through the diffusion network to obtain the first sample processing result, including: generating a specified number of preset noises through the diffusion network and adding them sequentially to the second sample image in the sample pair to obtain the first sample processing result;

[0053] The process involves using the diffusion network to perform back-diffusion processing on the first sample processing result, utilizing the sample mask image, sample image features, and sample text features, to obtain a predicted image. This includes: fusing the current sample fusion result with the sample mask image using the diffusion network to obtain a current first sample fusion result; wherein the sample fusion result used in the first fusion to obtain the first sample fusion result is the first sample processing result; processing the current first sample fusion result using the sample image features and sample text features based on a cross-attention mechanism using the diffusion network to obtain a current second sample processing result; denoising the current sample fusion result using the current second sample processing result using the diffusion network to obtain a new sample fusion result, and then returning to the step of fusing the current sample fusion result with the sample mask image using the diffusion network to obtain the current first sample fusion result, until the number of denoising operations reaches the specified number, resulting in a predicted image.

[0054] The model parameter adjustment module is specifically used to calculate the sub-loss value corresponding to each second sample processing result based on the difference between the second sample processing result and the corresponding preset noise. Specifically, the i-th second sample processing result obtained in the denoising order corresponds to the (n+1)-i-th added preset noise, where n represents a specified number. Based on the sub-loss value corresponding to each second sample processing result, a loss value is calculated. According to the calculated loss value, the model parameters of the initial structure image generation model are adjusted until a preset convergence condition is met, resulting in a trained image generation model.

[0055] Optionally, the image generation model of the initial structure further includes: a text feature extraction layer and an image feature extraction layer, and the diffusion network includes: an encoding layer, a decoding layer, a first multi-head attention layer and a second multi-head attention layer;

[0056] For each sample pair, image features are extracted from the first sample image to obtain sample image features, including:

[0057] The first sample image in each sample pair is input into the image feature extraction layer to obtain the sample image features;

[0058] The text feature extraction of the sample description text used to describe the second sample image in the sample pair is performed to obtain sample text features, including: inputting the sample description text used to describe the second sample image in the sample pair into the text feature extraction layer to obtain sample text features;

[0059] The diffusion network, based on a cross-attention mechanism, processes the current first sample fusion result using the sample image features and the sample text features to obtain the current second sample processing result, including:

[0060] The current first sample fusion result is encoded using the encoding layer to obtain the sample encoding result;

[0061] The first multi-head attention layer is used to process the obtained sample encoding result and the sample image features to obtain a first sample attention result. The second multi-head attention layer is used to process the obtained sample encoding result and the sample text features to obtain a second sample attention result. The first sample attention result and the second sample attention result are then fused to obtain a second sample fusion result.

[0062] The obtained second sample fusion result is decoded using the decoding layer to obtain the current second sample processing result.

[0063] Optionally, the text feature extraction layer is the part of the pre-trained image-text pre-training model used to extract text features, and the parameters of the text feature extraction layer remain unchanged during the training of the image generation model with the initial structure.

[0064] Optionally, the diffusion network is implemented based on Unet, and a first multi-head attention layer and a second multi-head attention layer are set between each encoding layer and its corresponding decoding layer; the query of the first multi-head attention layer and the second multi-head attention layer corresponding to each encoding layer is obtained based on the output data of that encoding layer; the data obtained by performing different linear transformations on the sample image features are respectively the key and value of the first multi-head attention layer; the data obtained by performing different linear transformations on the sample text features are respectively the key and value of the second multi-head attention layer.

[0065] Optionally, the device further includes:

[0066] The first image feature extraction module is used to extract features from the first image to be detected containing the preset type of object using the image feature extraction layer in the trained image generation model, thereby obtaining the first image features; wherein, the first image features are used to perform image detection on the first image to be detected.

[0067] Optionally, a sample pair is obtained based on the following steps: selecting any two images from pre-acquired images containing objects of the preset type and having the object's bounding box; transforming the two selected images so that the transformed images have the same size, and the bounding boxes of the two images have the same position and size in their respective images, thus obtaining a sample pair.

[0068] A fourth aspect of this application also provides an image generation apparatus, the apparatus comprising:

[0069] The reference image acquisition module is used to acquire a reference image containing objects of a preset type, and a bounding box representing the image area occupied by the objects of the preset type.

[0070] The image generation module is used to input the reference image, preset descriptive text, and preset noise image into a pre-trained image generation model to obtain an image to be used that conforms to the preset descriptive text and contains an object of the preset type; wherein, the image region occupied by the object of the preset type in the image to be used is the same in position and size as the image region occupied by the object of the preset type in the reference image; the preset noise image is the same size as the reference image; and the image generation model is trained based on any of the image generation model training methods described in the first aspect above.

[0071] Optionally, the image generation module is specifically used to perform the following steps through a pre-trained image generation model: extracting image features from the reference image to obtain image features to be used; binarizing the reference image according to the position of the bounding box in the reference image to obtain a mask image to be used; extracting text features from the preset descriptive text to obtain text features to be used; and using a diffusion network, performing backdiffusion processing on the preset noisy image using the mask image to be used, the image features to be used, and the text features to be used to obtain an image to be used that conforms to the preset descriptive text and contains an object of the preset type.

[0072] Optionally, the step of using a diffusion network to perform back-diffusion processing on the preset noisy image using the mask image to be used, the features of the image to be used, and the features of the text to be used to obtain a usable image that conforms to the preset descriptive text and contains an object of the preset type includes:

[0073] The current fusion result to be utilized is fused with the mask image to be utilized through a diffusion network to obtain the current first fusion result to be utilized; wherein, the fusion result to be utilized used when the first fusion is performed to obtain the first fusion result to be utilized is the preset noise image;

[0074] The diffusion network, based on the cross-attention mechanism, processes the current first fusion result to be utilized using the image features and text features to be utilized, thereby obtaining the current processing result to be utilized.

[0075] The diffusion network is used to denoise the current first fusion result to be utilized using the current processing result to obtain a new fusion result to be utilized. Then, the process returns to the step of fusing the current fusion result to be utilized with the mask image to be utilized using the diffusion network to obtain the current first fusion result to be utilized. This process continues until the number of denoising operations reaches a specified number, resulting in a fusion image to be utilized that conforms to the preset description text and contains objects of the preset type.

[0076] Optionally, the image generation model further includes a text feature extraction layer and an image feature extraction layer, and the diffusion network includes an encoding layer, a decoding layer, a first multi-head attention layer, and a second multi-head attention layer;

[0077] Image feature extraction is performed on the reference image to obtain the image features to be used, including:

[0078] The reference image is input into the image feature extraction layer to obtain the image features to be used;

[0079] Text feature extraction is performed on the preset descriptive text to obtain the text features to be utilized, including:

[0080] Input the preset descriptive text into the text feature extraction layer to obtain the text features to be utilized;

[0081] The diffusion network, based on a cross-attention mechanism, processes the current first fusion result using the image features and text features to be utilized, to obtain the current processing result, including:

[0082] The current first fusion result to be utilized is encoded using the encoding layer to obtain the encoded result to be utilized.

[0083] The first multi-head attention layer is used to process the obtained encoding result to be used and the image features to be used to obtain a first attention result to be used. The second multi-head attention layer is used to process the obtained encoding result to be used and the text features to be used to obtain a second attention result to be used. The first attention result to be used and the second attention result to be used are fused to obtain a second fused result to be used.

[0084] The second fusion result to be utilized is decoded using the decoding layer to obtain the current processing result to be utilized.

[0085] Optionally, the diffusion network is implemented based on Unet, and a first multi-head attention layer and a second multi-head attention layer are set between each encoding layer and its corresponding decoding layer; the query of the first multi-head attention layer and the second multi-head attention layer corresponding to each encoding layer is obtained based on the output data of the encoding layer, and the data obtained by performing different linear transformations on the image features to be utilized are respectively the key and value of the first multi-head attention layer; the data obtained by performing different linear transformations on the text features to be utilized are respectively the key and value of the second multi-head attention layer.

[0086] Optionally, the device further includes:

[0087] The second image feature extraction module is used to extract features from the second image to be detected containing the preset type of object using the image feature extraction layer, so as to obtain the features of the second image to be detected.

[0088] The image detection result determination module is used to input the features of the second image to be detected into the classification layer to obtain the image detection result of the second image to be detected.

[0089] In another aspect of the embodiments of this application, an electronic device is also provided, comprising:

[0090] Memory, used to store computer programs;

[0091] When a processor executes a program stored in memory, it implements the method described in either the first or second aspect above.

[0092] In another aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the method described in either the first aspect or the second aspect.

[0093] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods described in either the first or second aspect.

[0094] Beneficial effects of the embodiments in this application:

[0095] The image generation model training method provided in this application embodiment can acquire multiple sample pairs. Each sample pair includes two sample images of the same size, both containing objects of a preset type. Both sample images have bounding boxes representing the image area occupied by the objects, and the bounding boxes of the two sample images are in the same position and size. The first and second sample images in each sample pair, along with sample description text describing the second sample image in the sample pair, are input into an initial structure image generation model to obtain a predicted image output by the initial structure image generation model. Based on the difference between the predicted image obtained by the diffusion network in the initial structure image generation model and the second sample image in the sample pair when processing each sample pair, the model parameters of the initial structure image generation model are adjusted until a preset convergence condition is reached, thus obtaining a trained image generation model.

[0096] Based on the above processing, multiple sample pairs can be obtained, and the image generation model can be trained using two sample images and the descriptive text of one of the images in each pair. Since the descriptive text is used during the training process, the image generation model can learn the features of the descriptive text, thus ensuring that the output image matches the descriptive text. Furthermore, for each sample pair, since the two sample images are of the same size, both contain objects of a preset type, and the bounding boxes in both sample images have the same position and size, the image generation model can learn the features of the bounding box position and size in the image. This ensures that the position of the object in the output image matches the position of the object in the sample image, and the size of the image area occupied by the object in the output image matches the size of the area occupied by the object in the sample image. Thus, the image obtained by the image generation model matches the input descriptive text, and based on the position and size of the bounding boxes in the input image, the position and size of the object in the image obtained by the image generation model can be determined without manual determination and labeling of the object's position and size, reducing manual costs and improving the efficiency of obtaining sample images.

[0097] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0098] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0099] Figure 1This is a schematic diagram of the first process of the image generation model training method provided in the embodiments of this application;

[0100] Figure 2a A schematic diagram of a sample pair to be processed provided in an embodiment of this application;

[0101] Figure 2b A method for providing an embodiment of this application Figure 2a A schematic diagram of the sample pairs obtained by image transformation of the sample pairs to be processed;

[0102] Figure 3 This is a schematic diagram of the first process of image generation model generating an image in the image generation model training method provided in the embodiments of this application;

[0103] Figure 4 This is a schematic diagram of the second process of image generation by the image generation model in the image generation model training method provided in the embodiments of this application;

[0104] Figure 5 This is a schematic diagram of the third process of image generation model generating images in the image generation model training method provided in the embodiments of this application;

[0105] Figure 6 A schematic diagram of the structure of a text-image pre-training model provided in an embodiment of this application;

[0106] Figure 7 A schematic diagram of a diffusion network provided in an embodiment of this application;

[0107] Figure 8 This is a schematic diagram of a second process for training an image generation model provided in an embodiment of this application;

[0108] Figure 9 A schematic diagram of the third process of the image generation model training method provided in the embodiments of this application;

[0109] Figure 10 A flowchart illustrating an image generation method provided in an embodiment of this application;

[0110] Figure 11 This is a schematic diagram of the structure of an image generation model training device provided in an embodiment of this application;

[0111] Figure 12 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application;

[0112] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0113] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0114] With the development of computer vision technology, its application scenarios are becoming increasingly widespread, such as target (e.g., human faces or animal) detection. In target detection, deep learning models can be used. To ensure the model's generalization ability, a large number of sample images are often needed to train the model; therefore, the training sample set needs to be expanded. Related technologies can generate images using pre-trained sample generation models, such as GAN or Diffusion. Subsequently, the objects to be detected (referred to as the objects to be detected) in the generated images need to be manually labeled, which consumes significant manual labor and is inefficient, resulting in low efficiency in acquiring sample images.

[0115] To reduce labor costs and improve the efficiency of acquiring sample images, this application provides an image generation model training method, see [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of a first flowchart of an image generation model training method provided in an embodiment of this application. The method includes:

[0116] Step S101: Obtain multiple sample pairs.

[0117] Each sample pair includes two sample images of the same size, each containing an object of a preset type. Both sample images have bounding boxes indicating the image area occupied by the object, and the bounding boxes of the two sample images are in the same position and size.

[0118] Step S102: Input the first sample image and the second sample image in each sample pair, as well as the sample description text used to describe the second sample image in the sample pair, into the image generation model of the initial structure to obtain the predicted image output by the image generation model of the initial structure.

[0119] Step S103: Based on the difference between the predicted image obtained by the diffusion network in the initial image generation model and the second sample image in the sample pair when processing each sample pair, adjust the model parameters of the initial image generation model until the preset convergence condition is reached, and obtain the trained image generation model.

[0120] Based on the above processing, multiple sample pairs can be obtained, and the image generation model can be trained using two sample images and the descriptive text of one of the images in each pair. Since the descriptive text is used during the training process, the image generation model can learn the features of the descriptive text, thus ensuring that the output image matches the descriptive text. Furthermore, for each sample pair, since the two sample images are of the same size, both contain objects of a preset type, and the bounding boxes in both sample images have the same position and size, the image generation model can learn the features of the bounding box position and size in the image. This ensures that the position of the object in the output image matches the position of the object in the sample image, and the size of the image area occupied by the object in the output image matches the size of the area occupied by the object in the sample image. Thus, the image obtained by the image generation model matches the input descriptive text, and based on the position and size of the bounding boxes in the input image, the position and size of the object in the image obtained by the image generation model can be determined without manual determination and labeling of the object's position and size, reducing manual costs and improving the efficiency of obtaining sample images.

[0121] In step S101, multiple sample pairs consisting of two identical sample images can be obtained. Each sample pair contains two images that will be detected later (i.e., objects of a preset type), and each image has a bounding box representing the image region occupied by the object. The bounding boxes in the two images are identical in position and size. For example, the position of the bounding boxes in each image can be represented by the pixel coordinates of a vertex or center point of the bounding box, and the size of the bounding box can be represented by its length and width. In other words, the objects of the preset type occupy the same image region in the two sample images belonging to the same sample pair, and their sizes are also the same. For example, the objects of the preset type can be a human face, a vehicle, or an animal. Subsequently, using each sample pair, the image generation model of the initial structure is trained, enabling the image generation model to learn the features of the bounding box positions in the image, thus ensuring that the position of the object in the output image matches the position of the object in the sample images.

[0122] In one implementation, images with labeled locations of objects of a preset type in each image can be searched from the network, or images with labeled locations of objects of a preset type in each image can be obtained from a publicly available dataset to obtain multiple sample pairs.

[0123] In another implementation, multiple sample pairs can be obtained from pre-collected images containing objects of a preset type. For example, multiple images containing objects of a preset type can be pre-acquired, and the positions of the objects in each image can be determined and labeled, resulting in images with bounding boxes representing the image regions occupied by the objects of the preset type. Correspondingly, the position of the bounding box in the image can represent the position of the image region occupied by the object of the preset type. Using the labeled images, multiple sample pairs can be constructed, each consisting of two sample images of the same size, where the bounding boxes of the two sample images in each pair have the same position and size within their respective images.

[0124] In one embodiment, a sample pair is obtained based on the following steps: selecting any two images from a pre-acquired image set containing objects of a preset type and with object bounding boxes; transforming the two selected images so that the transformed images are the same size and the bounding boxes of the two images are in the same position and size in their respective images, thus obtaining a sample pair.

[0125] In this embodiment, images containing objects of a preset type and with bounding boxes (i.e., annotated images) can be pre-obtained. Any two images (referred to as a pair of images to be processed) can be selected from the annotated images. By performing image transformation on the selected two images, two transformed images of the same size can be obtained, and the positions and sizes of the bounding boxes in the two transformed images are identical. That is, the scale information of the two transformed images, as well as the position and scale information of their bounding boxes, are consistent. The two transformed images can then be considered as a sample pair.

[0126] For example, image transformation can include cropping and / or scaling, without specific limitations, as long as it can produce two images with consistent bounding box position and scale information, and consistent scale information themselves. For instance, one image in a pair of images to be processed can be cropped, and based on the cropped image, the other image in the pair can be cropped and / or scaled so that the processed image has the same size as the cropped image, and the position and scale information of the bounding boxes of the processed image are consistent with the cropped image.

[0127] Based on the above processing, image transformation can be used to process the annotated images, resulting in multiple sample pairs, each consisting of two identical sample images. The bounding boxes in each sample pair have the same position and size within their respective images. Subsequently, the initial image generation model is trained using these sample pairs. This allows the model to learn the features of the bounding box position and size within the image, ensuring that the position of objects in the output image matches the position of objects in the sample images, and that the size of the image region occupied by the objects in the output image matches that of the objects in the sample images. Thus, the image generated by the image generation model matches the input descriptive text. Furthermore, the position and size of objects in the image generated by the image generation model can be determined based on the position and size of the bounding boxes in the input image, eliminating the need for manual determination and annotation of object positions and sizes. This reduces manual labor costs and improves the efficiency of acquiring sample images.

[0128] In one embodiment, the object of the preset type is a human face. See also Figure 2a , Figure 2a This is a schematic diagram of a sample pair to be processed, provided as an embodiment of this application. Figure 2a The two images in the image represent the sample pair to be processed. Both images contain human faces and have bounding boxes indicating the area occupied by the faces. The bounding boxes representing the edges of the black areas in both images are their respective bounding boxes. To protect privacy, the bounding boxes in the images are filled with black. Understandably, in the actual processing, the bounding boxes in the two images are not filled.

[0129] See Figure 2b , Figure 2b A method for providing an embodiment of this application Figure 2a The diagram shows a sample pair obtained by image transformation of the sample pair to be processed. Figure 2b The two images in the dataset are the same size, and the bounding boxes in both images are located in the same position within their respective datasets, with the bounding boxes also being the same size. It should be noted that the images containing human faces in this embodiment are from publicly available datasets.

[0130] Regarding step S102, the first sample image can be any one of the images in the sample pair, and correspondingly, the second sample image is the other image in the sample pair. For each sample pair, the two sample images in the pair, along with the descriptive text used to describe the second sample image (i.e., the sample description text), can be input into the image generation model of the initial structure. For example, the description text of a sample image can be used to describe the attribute information of a predefined type of object in the sample image, such as... Figure 2bThe image on the left is the first sample image, and the image on the right is the second sample image. The image on the right is a frontal photograph of a young woman, and its descriptive text could be "an image of a young woman's face".

[0131] The initial image generation model may include a diffusion network. Accordingly, the initial image generation model can output a predicted image based on the input sample image and descriptive text through the diffusion network. The data processing by the diffusion network may include a forward diffusion process and a backward diffusion process. In the forward diffusion process, noise can be gradually added to the data to obtain noisy data. For details, please refer to the relevant descriptions of steps S304 and S3041 in subsequent embodiments. In the backward diffusion process, the noise added at each step can be predicted, and the noise in the noisy data can be gradually removed based on the predicted noise to recover the original high-quality data from the noisy data and obtain the predicted image. For details, please refer to the relevant descriptions of steps S305 and steps S3051-S3053 in subsequent embodiments.

[0132] Subsequently, the loss value can be calculated by comparing the noise added by the diffusion network when obtaining the predicted image with the predicted noise for each sample pair. The model parameters of the initial image generation model can then be adjusted based on this loss value. Since descriptive text is used during the training of the image generation model, it learns the features of the descriptive text, thus enabling the output image to conform to that descriptive text.

[0133] In addition, for each sample pair, since the two sample images are the same size, both contain objects of a preset type, and the bounding boxes of the two sample images are in the same position and size, the image generation model can learn the features of the bounding box position and size in the image. This allows the position of the object in the output image to be consistent with the position of the object in the sample image, and the size of the image area occupied by the object in the output image is also consistent with the size of the area occupied by the object in the sample image.

[0134] Thus, the ability of the image generation model to generate images that match the input descriptive text and contain the features of the input images can be trained using the two sample images included in each sample pair and the descriptive text of one of the sample images in that sample pair. The input image features are the most significant commonalities between the two images in each input sample pair, namely, that both images contain objects of a predefined type, and the position of the objects in each image and the size of the image regions occupied by the objects in each image are the same.

[0135] Regarding step S103, the model parameters of the initial image generation model can be adjusted based on the difference between the predicted image obtained by the diffusion network in the initial image generation model and the second sample image in the sample pair when processing each sample pair, until a preset convergence condition is met, resulting in a trained image generation model. For example, the loss value can be calculated based on the difference between the predicted image obtained by the diffusion network in the initial image generation model and the second sample image in the sample pair when processing each sample pair. During the loss value calculation, the difference between the noise added by the diffusion network when obtaining the predicted image from a sample pair and the predicted noise can be used for calculation. Correspondingly, the loss value for each sample pair can be obtained. Furthermore, the model parameters of the initial image generation model can be adjusted based on the calculated loss value. For example, the model parameters of the initial image generation model can be adjusted based on the loss value corresponding to a batch of sample pairs. A batch can contain a preset number of sample pairs, which can be set as needed and is not specifically limited. The preset convergence condition can be that the calculated loss value is less than a preset threshold.

[0136] Thus, during the training process of the image generation model, the model can output a predicted image that more closely resembles the image described in the descriptive text, based on the two images in each sample pair and the descriptive text describing one of the images in that pair. Because the descriptive text is used during training, the image generation model learns the features of the descriptive text, enabling the output image to match the descriptive text. Furthermore, for each sample pair, since the two sample images are of the same size, both contain objects of a preset type, and the bounding boxes in both sample images have the same position and size, the image generation model learns the features of the bounding box position and size in the image. This ensures that the position of the object in the output image matches the position of the object in the sample image, and the size of the image region occupied by the object in the output image matches the size of the region occupied by the object in the sample image.

[0137] Subsequently, using the trained image generation model, a new image containing objects of the preset types can be generated based on a small number of pre-annotated images containing the locations of objects of the preset types, combined with the descriptive text for the image to be generated. Furthermore, the generated image conforms to the input descriptive text, and based on the position and size of the bounding boxes in the input image, the position and size of the objects in the image obtained by the image generation model can be determined.

[0138] In one embodiment, see Figure 3 , Figure 3This is a schematic diagram of the first process of generating an image using the image generation model in the image generation model training method provided in this application embodiment. Step S102 includes: performing the following steps using the image generation model with the initial structure:

[0139] Step S301: Extract image features from the first sample image in each sample pair to obtain sample image features.

[0140] Step S302: Binarize the first sample image according to the position of the marked box in the first sample image to obtain the sample mask image.

[0141] Step S303: Extract text features from the sample description text used to describe the second sample image in the sample pair to obtain sample text features.

[0142] Step S304: Perform forward diffusion processing on the second sample image in the sample pair using a diffusion network to obtain the processing result of the first sample.

[0143] Step S305: Using a diffusion network, the first sample processing result is back-diffused with the sample mask image, sample image features, and sample text features to obtain the predicted image.

[0144] In this embodiment, steps S301-S305 can be performed using an image generation model with an initial structure. Specifically, feature extraction can be performed on the first sample image in each sample pair to obtain the image features (i.e., sample image features) of the first sample image in the sample pair. The specific method for extracting sample image features can be found in the description of step S3011 in subsequent embodiments. Feature extraction is then performed on the sample description text used to describe the second sample image in the sample pair to obtain the text features (i.e., sample text features) of the sample description text. The specific method for extracting sample text features can be found in the description of step S3031 in subsequent embodiments.

[0145] In a sample image, the region represented by the bounding box is the region occupied by the object of the preset type within that sample image. That is, the position of the bounding box in the first sample image is the position of the region occupied by the object of the preset type. For example, the position of the bounding box can be represented by the pixel coordinates of its four corners. Binarizing the first sample image according to the position of the bounding box allows us to set the pixel values ​​of all pixels in the region occupied by the object of the preset type to the same value, and set the pixel values ​​of all pixels in the region occupied by objects that are not of the preset type to a different value, thus obtaining a sample mask image. For example, in the sample mask image, the pixel value of the region occupied by the object of the preset type is 255 (i.e., white), and the pixel value of all other pixels is 0 (i.e., black). The sample mask image can be obtained using the following formula:

[0146] M ia =Binary rect (ImageA)

[0147] Among them, M ia Represents the sample mask image, Binary rect This indicates that the image is being binarized, and ImageA represents the first sample image.

[0148] In the diffusion network, forward diffusion processing can be performed, that is, the diffusion network adds preset noise to the second sample image of the sample pair to obtain the first sample processing result. For example, the preset noise can be obtained by randomly sampling Gaussian noise that conforms to a standard normal distribution. In one implementation, there can be multiple preset noises, each of which is obtained by random sampling. Accordingly, the diffusion network can add preset noise multiple times based on the second sample image. After all preset noises have been added, the first sample processing result can be obtained. The specific process of adding preset noise can be referred to the relevant description of step S3041 in the subsequent embodiments.

[0149] After obtaining the first sample processing result, the diffusion network can use the sample mask image, sample image features, and sample text features to perform back-diffusion processing on the first sample processing result to obtain the predicted image. That is, the diffusion network uses the sample mask image, sample image features, and sample text features to denoise the first sample processing result multiple times to obtain the predicted image. For example, the diffusion network can use the sample mask image, sample image features, sample text features, and the first sample processing result to obtain multiple prediction noises, and can use the obtained prediction noises to denoise the first sample processing result multiple times. After all the prediction noises are removed, the predicted image is obtained. For details, please refer to the relevant descriptions of steps S3051-S3053 in the subsequent embodiments.

[0150] Based on the above processing, by executing steps S301-S305 through the image generation model with the initial structure, a predicted image can be obtained. Furthermore, sample text features are used during training, enabling the image generation model to learn the features of the descriptive text, thus ensuring that the output image matches the descriptive text. Additionally, for each sample pair, since the two sample images are of the same size, both contain objects of a preset type, and the bounding boxes in the two sample images have the same position and size, the image generation model can learn the features of the sample images—that is, the features of the position and size of the bounding boxes in the sample images. This allows the position of the object in the output image to be consistent with the position of the object in the sample image, and the size of the image region occupied by the object in the output image is also consistent with the size of the region occupied by the object in the sample image.

[0151] In this way, the image obtained by the image generation model can match the input descriptive text, and the position and size of the object in the image obtained by the image generation model can be determined based on the position and size of the bounding box in the input image, without the need for manual determination and labeling of the object's position and size, which can reduce labor costs and improve the efficiency of obtaining sample images.

[0152] In one embodiment, see Figure 4 , Figure 4 This is a schematic diagram of a second process for generating images using the image generation model in the image generation model training method provided in this application embodiment. Step S304 includes: Step S3041: Generating a specified number of preset noises through a diffusion network and sequentially adding them to the second sample image of the sample pair to obtain the first sample processing result.

[0153] Step S305 includes: Step S3051: The current sample to be fused result is fused with the sample mask image through a diffusion network to obtain the current first sample fusion result.

[0154] The sample to be fused used in the first fusion to obtain the first sample fusion result is the first sample processing result.

[0155] Step S3052: Using a diffusion network based on a cross-attention mechanism, the current first sample fusion result is processed using sample image features and sample text features to obtain the current second sample processing result.

[0156] Step S3053: Denoise the current sample to be fused result using the current second sample processing result through the diffusion network to obtain a new sample to be fused result, and return to execute step S3051 until the number of denoising times reaches the specified number to obtain the predicted image.

[0157] Step S103 includes:

[0158] Step 1: For each second sample processing result, calculate the sub-loss value corresponding to the second sample processing result based on the difference between the second sample processing result and the corresponding preset noise.

[0159] The i-th second sample processing result obtained in the order of denoising corresponds to the (n+1-i)-th preset noise added, where n represents the specified number.

[0160] Step 2: Calculate the loss value based on the sub-loss value corresponding to the processing result of each second sample.

[0161] Step 3: Based on the calculated loss value, adjust the model parameters of the initial structure image generation model until the preset convergence condition is met, and obtain the trained image generation model.

[0162] In this embodiment, a specified number of preset noises can be generated. For example, a specified number of random samples can be performed on Gaussian noise conforming to a standard normal distribution to obtain a specified number of preset noises. These preset noises can then be added to the second sample image in the sample pair to obtain the first sample processing result (also called the sample noise image). Specifically, the first preset noise can be added to the second sample image to obtain a noisy image; the second preset noise can be added to the obtained noisy image to obtain a new noisy image; the third preset noise can be added to the new noisy image to obtain a latest noisy image, and so on. This continues until the last preset noise is added, resulting in a sample noise image (which can be called the first noise image). The first noise image is the first sample noise image to be denoised subsequently. This process is the forward diffusion process of the diffusion network, gradually adding preset noise to the image.

[0163] The first sample fusion result is obtained by fusing the latest sample fusion result with the sample mask image using a diffusion network. The sample fusion result used in the first fusion to obtain the first sample fusion result is the first sample processing result. For example, a cat (concatenate) operation can be performed on the latest sample fusion result and the sample mask image to obtain the current first sample fusion result; alternatively, a weighted sum of the latest sample fusion result and the sample mask image can be calculated to obtain the current first sample fusion result.

[0164] Since the sample mask image is obtained by binarizing the first sample image according to the positions of the bounding boxes in the first sample image, the sample mask image can contain the positional information of the image region occupied by the object of the preset type. Therefore, the first sample fusion result obtained by fusing the current sample to be fused result with the sample mask image can contain the positional information of the image region occupied by the object of the preset type.

[0165] Furthermore, using a diffusion network based on a cross-attention mechanism, the current first sample fusion result is processed using sample image features and sample text features to obtain the current second sample processing result (which can be called prediction noise). There are multiple prediction noises, the same number as the preset noise added in step S3041, both being a specified number, and each prediction noise corresponds one-to-one with the preset noise. Using the obtained multiple prediction noises, denoising can be performed multiple times based on the first sample processing result. After each denoising, a new sample noise image is obtained, which serves as the current sample noise image (i.e., the current sample fusion result). After all multiple prediction noises have been removed, the prediction image is obtained.

[0166] Steps S3051 and S3052 process the first sample processing result to obtain a second sample processing result (which can be called the first noise to be removed), which is the predicted noise (which can be called the first noise to be removed) corresponding to the last added preset noise. The first noise to be removed is the first noise to be removed. Denoising the first noise image using the first noise to be removed yields a new sample fusion result, i.e., a new sample noise image. Step S3051 can then be returned to be executed to process this new sample noise image using steps S3051 and S3052, predicting the second-to-last added noise. The resulting second sample processing result is the predicted noise (which can be called the second noise to be removed) corresponding to the second-to-last added preset noise. Denoising the new sample noise image using the second noise to be removed yields a latest sample noise image, and step S3051 can be returned to be executed, and so on. The denoising process continues until a specified number of times the noisy sample image is denoised. This indicates that all the specified number of pre-defined noises have been predicted, resulting in corresponding predicted noises that have been removed from the noisy sample image. In other words, denoising is complete, and the predicted image can be obtained after denoising. Specifically, the i-th predicted noise obtained in the denoising order corresponds to the (n+1)-i-th added pre-defined noise, where n represents the specified number. This process is the back-diffusion process of the diffusion network, gradually removing noise from the noisy data (i.e., the first sample processing result) to recover high-quality data and obtain the predicted image.

[0167] For each second sample processing result (i.e., predicted noise), a sub-loss value corresponding to the predicted noise can be calculated based on the difference between the predicted noise and the corresponding preset noise. For example, the difference between the predicted noise and the corresponding preset noise, or the square of the 2-norm, can be calculated to obtain the sub-loss value of the predicted noise. For each sample pair, when the image generation model is trained using that sample pair, a specified number of predicted noises and preset noises can be obtained. Each predicted noise can have a corresponding sub-loss value, and consequently, a specified number of sub-loss values ​​can be obtained. After obtaining the sub-loss values ​​corresponding to each predicted noise, the loss value can be calculated using each sub-loss value. For example, the average or expected value of each sub-loss value can be calculated to obtain the loss value. For example, the loss value can be calculated using the following formula:

[0168]

[0169] in, This indicates the expectation of the function enclosed in square brackets; y0 represents the square of the 2-norm; y0 represents the second sample image; ∈ represents the preset noise obtained by random sampling from Gaussian noise that conforms to the standard normal distribution, i.e., ∈ ∈ N(0,1); ∈θ represents the predicted noise output by the diffusion network; t ∈ U[1,T] represents the time step of sampling from the uniform distribution U, and the preset noise is added once at every t time. This represents the image yt (i.e., the sample noise image) obtained by injecting noise into y0 and obtaining the image at time t. The product of scaling factors at various times can be represented by... It means that α s α represents the scaling factor at time s; s =1-β s ,β s M represents the pre-set noise variance at time s; ia denoted as the sample mask image (also known as the binary mask), and c represents the guiding conditions of the diffusion network, namely the sample text features and sample image features.

[0170] Based on the above processing, a specified number of preset noises can be gradually added to a second sample image in a sample pair using a diffusion network, resulting in a sample noise image (i.e., the first sample processing result). Prediction is performed on the preset noise added at each step to obtain the corresponding predicted noise (i.e., the second sample processing result), and the sample noise image is progressively denoised based on the obtained predicted noise to obtain the predicted image. This allows the image generation model to learn the ability to progressively denoise and generate images from noisy images, ensuring that the required images can be generated subsequently using the image generation model. During the denoising process, the model learns the features of the descriptive text and the features of the bounding boxes' positions in the image, ensuring that the output image matches the descriptive text and that the positions of objects in the output image are consistent with the positions of objects in the sample images. Furthermore, using the trained image generation model, a new image containing objects of the preset types can be generated based on a small number of already labeled images containing the positions of objects of the preset types, combined with the descriptive text for which the image to be generated needs to be created. The generated image matches the input descriptive text, and the positions of objects in the image obtained by the image generation model can be determined based on the positions of the bounding boxes in the input image.

[0171] In one embodiment, see Figure 5 , Figure 5 This is a schematic diagram illustrating the third process of image generation in the image generation model training method provided in this application embodiment. The initial structure of the image generation model further includes: a text feature extraction layer and an image feature extraction layer, and the diffusion network includes: an encoding layer, a decoding layer, a first multi-head attention layer, and a second multi-head attention layer;

[0172] Step S301 includes: Step S3011: Input the first sample image in each sample pair into the image feature extraction layer to obtain sample image features.

[0173] Step S303 includes: Step S3031: Input the sample description text used to describe the second sample image in the sample pair into the text feature extraction layer to obtain sample text features.

[0174] Step S3052 includes:

[0175] Step S30521: Use the coding layer to encode the current first sample fusion result to obtain the sample coding result.

[0176] Step S30522: Using the first multi-head attention layer, process the obtained sample encoding results and sample image features to obtain the first sample attention result; and using the second multi-head attention layer, process the obtained sample encoding results and sample text features to obtain the second sample attention result; and fuse the first sample attention result and the second sample attention result to obtain the second sample fusion result.

[0177] Step S30523: Use the decoding layer to decode the obtained second sample fusion result to obtain the current second sample processing result.

[0178] In this embodiment, the image generation model may include a text feature extraction layer, an image feature extraction layer, and a diffusion network. For example, the image feature extraction layer may be a CNN (Convolutional Neural Networks) or a transformer (a network architecture), and the text feature extraction layer may be the part of CLIP (Contrastive Language-Image Pre-training, a text and image pre-training model, i.e., the image-text pre-training model in subsequent embodiments) used for extracting text features, BERT (a language representation model), or LSTM (Long Short-Term Memory). The diffusion network may include an encoding layer, a decoding layer, a first multi-head attention layer, and a second multi-head attention layer. For example, the first and second multi-head attention layers may be the MHA (Multi-Head Attention) module in a transformer model.

[0179] The text feature extraction layer can be used to extract features from the sample description text, thus obtaining the sample text features. For example, this can be represented by the following formula:

[0180] F t =CLIPtext (text-prompt)

[0181] Among them, F t CLIP represents the features of the sample text. text This indicates that text features are extracted through the text feature extraction layer, which is the part of CLIP used to extract text features. text-prompt represents the sample description text.

[0182] Image features are extracted from the first sample image using an image feature extraction layer to obtain the sample image features. For example, this can be expressed by the following formula:

[0183] F ia =Encoder img (ImageA)

[0184] Among them, F ia Encoder represents the features of the sample image. img This indicates that image features are extracted through the image feature extraction layer, and ImageA represents the first sample image.

[0185] Sample image features can be used as input to the first multi-head attention layer, and sample text features can be used as input to the second multi-head attention layer. The first sample fusion result can contain the location information of the bounding boxes in the image. Correspondingly, the first sample fusion result is encoded by an encoding layer, and the resulting encoded sample result can also contain the location information of the bounding boxes in the image. The encoded sample result can be input into the first and second multi-head attention layers for processing.

[0186] A first multi-head attention layer processes the obtained sample encoding results and sample image features to obtain a first sample attention result. A second multi-head attention layer processes the obtained sample encoding results and sample text features to obtain a second sample attention result. The first and second sample attention results can then be fused to obtain a second sample fusion result. Correspondingly, the second sample fusion result can include features of the bounding box positions in the sample image and features of the descriptive text. A decoding layer decodes the second sample fusion result to obtain the current second sample processing result.

[0187] Based on the above processing, the image generation model can learn the features of the sample text, that is, the features of the descriptive text, thus enabling the output image to match the descriptive text. Furthermore, for each sample pair, since the two sample images are of the same size, both contain objects of a preset type, and the bounding boxes in the two sample images have the same position and size, the image generation model can learn the features of the bounding box position and size in the image. This ensures that the position of the object in the output image matches the position of the object in the sample image, and the size of the image area occupied by the object in the output image matches the size of the area occupied by the object in the sample image. Thus, the image obtained by the image generation model matches the input descriptive text, and based on the position and size of the bounding boxes in the input image, the position and size of the object in the image obtained by the image generation model can be determined without manual determination and labeling of the object's position and size, reducing labor costs and improving the efficiency of acquiring sample images.

[0188] In one embodiment, the text feature extraction layer is the part of the pre-trained image-text pre-training model used to extract text features, and the parameters of the text feature extraction layer remain unchanged during the training of the image generation model with the initial structure.

[0189] In this embodiment, multiple image-text pairs can be pre-acquired, each pair containing a text and an image, where the text describes the image. The initial image-text pre-training model is trained using these pairs to obtain a trained image-text pre-training model. The image-text pre-training model can include parts for extracting image features and parts for extracting text features. During training, the image-text pre-training model can learn the alignment relationship between the image and the text, ensuring that the text features extracted through the text feature extraction part of the image-text pre-training model reflect the image features described by the text. For example, the image-text pre-training model can be a CLIP.

[0190] like Figure 6 As shown, Figure 6 This is a schematic diagram of the structure of a text-image pre-training model provided in an embodiment of this application. During the training of the text-image pre-training model, for a given text-image pair, the image in the pair is input into the image encoder of the initial text-image pre-training model to obtain the image features. The text in the pair is then input into the text encoder of the initial text-image pre-training model to obtain the text features describing the image. Furthermore, a loss value can be calculated based on the difference between the image features and the text features describing the image, thereby adjusting the model parameters of the initial text-image pre-training model until the model convergence condition is met, resulting in a trained text-image pre-training model.

[0191] Since the image-text pre-trained model is trained with a large number of image and text pairs, in order to avoid destroying the text feature extraction layer's ability to extract text features and to ensure that the text features extracted by the text feature extraction layer can reflect the image features of the image described by the input descriptive text, the parameters of the text feature extraction layer are frozen during the training of the image generation model with the initial structure. That is, the parameters of the text feature extraction layer are not adjusted, only the parameters of the image feature extraction layer and the diffusion network are adjusted.

[0192] Based on the above processing, it can be ensured that the sample text features extracted by the text feature extraction layer can effectively reflect the image features of the image described by the input descriptive text. In turn, it can ensure that the image generation model can learn the features of the descriptive text so that the output image conforms to the descriptive text.

[0193] Furthermore, for each sample pair, since the two sample images are of the same size, both contain objects of a preset type, and the bounding boxes in both sample images have the same position and size, the image generation model can learn the features of the bounding box position and size in the image. This allows the position of the object in the output image to match the position of the object in the sample image, and the size of the image area occupied by the object in the output image is also consistent with the size of the area occupied by the object in the sample image. Thus, the image obtained by the image generation model can match the input descriptive text, and based on the position and size of the bounding boxes in the input image, the position and size of the object in the image obtained by the image generation model can be determined, eliminating the need for manual determination and labeling of the object's position and size. This reduces manual costs and improves the efficiency of acquiring sample images.

[0194] In one embodiment, the diffusion network is implemented based on Unet, and a first multi-head attention layer and a second multi-head attention layer are set between each encoding layer and its corresponding decoding layer; the query of the first multi-head attention layer and the second multi-head attention layer corresponding to each encoding layer is obtained based on the output data of the encoding layer; the data obtained by performing different linear transformations on the sample image features are respectively the key and value of the first multi-head attention layer; the data obtained by performing different linear transformations on the sample text features are respectively the key and value of the second multi-head attention layer.

[0195] In this embodiment, the diffusion network may include multiple encoding layers and multiple decoding layers, with a one-to-one correspondence between the encoding and decoding layers. A first multi-head attention layer and a second multi-head attention layer are configured between each encoding layer and its corresponding decoding layer. Based on the output data of each encoding layer, the query of the corresponding first and second multi-head attention layers can be obtained. For example, a linear transformation can be performed on the output data of each encoding layer to obtain the query of the corresponding first and second multi-head attention layers. Data obtained by performing different linear transformations on sample image features are respectively the key and value of the first multi-head attention layer; data obtained by performing different linear transformations on sample text features are respectively the key and value of the second multi-head attention layer.

[0196] The output data of each coding layer can be obtained using the following formula:

[0197] Q ib =Unet(Cat(Noise,M ia ))

[0198] Among them, Q ib M represents the output data of each coding layer (also known as the intermediate feature layer features of Unet), Noise represents the current sample noise image, that is, the current sample to be fused result, M ia Represents the sample mask image, Cat(Noise,M) ia This indicates the first sample fusion result obtained by fusing the current sample noise image and the sample mask image.

[0199] The result of the second sample fusion can be obtained using the following formula:

[0200] Q ct =MHA(Q ib ,K ia V ia )+MHA(Q ib ,K b-t V b-t )

[0201] Among them, Q ct This indicates the result of the second sample fusion; MHA(Q) ib ,K ia V ia ) represents the attention result of the first sample, Q ib K represents the output data of each coding layer. ia and V ia These represent the Key and Value of the first multi-head attention layer, respectively; MHA(Q) ib ,K b-t Vb-t ) represents the attention result of the second sample, K b-t and V b-t These represent the Key and Value of the second multi-head attention layer, respectively.

[0202] Taking the attention result of the first sample as an example, it is calculated using the following formula:

[0203]

[0204] Among them, MHA(Q) ib ,K ia V ia ) represents the attention result of the first sample, Q ib K represents the output data of each coding layer. ia and V ia These represent the Key and Value of the first multi-head attention layer, respectively; softmax represents the normalized exponential function. This indicates the pre-set scaling factor.

[0205] See Figure 7 , Figure 7 This is a schematic diagram of a diffusion network structure provided in an embodiment of this application. The diffusion network includes: an encoding part on the left, a decoding part on the right, and a designated network layer between the encoding part and the decoding part. Figure 7 (Not shown in the diagram). The encoding part includes multiple encoding layers and downsampling layers, and the decoding part includes multiple decoding layers and upsampling layers. The number of encoding layers and decoding layers is the same, and there is a one-to-one correspondence between the multiple encoding layers and the multiple decoding layers. The number of downsampling layers and upsampling layers is the same, and there is a one-to-one correspondence between the multiple downsampling layers and the multiple upsampling layers. In this embodiment, only 3 encoding layers and decoding layers, and 2 downsampling layers and upsampling layers are used as an example for illustration, but it is not limited to this.

[0206] Each encoding layer and its corresponding decoding layer are connected by a first multi-head attention layer (which can be simply referred to as the first attention layer) and a second multi-head attention layer (which can be simply referred to as the second attention layer). Figure 7 In this architecture, each encoding and decoding layer can be implemented using convolution. The output of each encoding layer passes through a first multi-head attention layer and a second multi-head attention layer to obtain the attention result. The plus sign indicates that the attention results from the first and second multi-head attention layers are fused. The fused result can be input to the corresponding decoding layer of the encoding layer, as well as to the next downsampling layer of the encoding layer. The output of the downsampling layer can be input to subsequent encoding layers and the corresponding upsampling layers. The output of each decoding layer can be input to the next upsampling layer.

[0207] In one embodiment, see Figure 8 , Figure 8This is a schematic diagram of a second process for training an image generation model provided in an embodiment of this application.

[0208] The method also includes:

[0209] Step S104: Using the image feature extraction layer in the trained image generation model, perform feature extraction on the first image to be detected containing objects of a preset type to obtain the features of the first image to be detected.

[0210] The first image feature to be detected is used to perform image detection on the first image to be detected.

[0211] In this embodiment, during the training of the image generation model, for each sample pair, since the two sample images are of the same size, both contain objects of a preset type, and the bounding boxes of the two sample images are in the same position and size, the image feature extraction layer of the image generation model can learn to extract the most significant common features between the two images in the sample pair. That is, both images contain objects of a preset type, the objects of the preset type are in the same position, and the size of the image region occupied by the objects is also the same. Therefore, the image feature extraction layer can learn to extract the features of the preset type of objects as well as the features of the position and size of the image region occupied by the objects in the image.

[0212] Furthermore, using the image feature extraction layer in the trained image generation model, features can be extracted from the first image to be detected, which contains objects of a preset type. The extracted features can include: the features of the preset type of objects in the first image to be detected, and the position and size of the image region occupied by the preset type of objects in the first image to be detected. Correspondingly, the extracted features can be used for subsequent image detection on the first image to be detected, to detect the preset type of objects contained in the first image to be detected. For example, the position of the preset type of objects in the first image to be detected can be detected, and the corresponding detection result can include: the position of the image region occupied by the preset type of objects in the first image to be detected; and / or, the category of the preset type of objects in the first image to be detected can be detected, and the corresponding detection result can include: the probability that the preset type of objects in the first image to be detected belong to each category.

[0213] Based on the above processing, the image feature extraction layer in the trained image generation model can also be applied to subsequent image detection tasks, and the trained image generation model can be used to generate images. In other words, based on the image generation model training method provided in this application embodiment, an image generation model for generating images can be obtained, and the image feature extraction layer in the trained image generation model can also be applied to subsequent image detection tasks. That is, the image feature extraction layer in the image detection task and the image generation model used for sample generation can be trained end-to-end simultaneously. Thus, using the trained image generation model, an image matching the input descriptive text can be obtained, and based on the position and size of the bounding boxes in the input image, the position and size of the objects in the image can also be obtained. This eliminates the need for manual determination and labeling of the object's position and size, reducing manual costs and improving the efficiency of obtaining sample images. Simultaneously, an image feature extraction layer applicable to subsequent image detection tasks can also be obtained.

[0214] In one embodiment, see Figure 9 , Figure 9 This is a schematic diagram of a third process for training an image generation model provided in an embodiment of this application. The image generation model may include: an image encoder (i.e., the image feature extraction layer in the above embodiment), a frozen CLIP text encoder (i.e., the text feature extraction layer in the above embodiment), and a denoiser (i.e., the diffusion network in the above embodiment).

[0215] Images A and B belong to a sample pair, where image A is the first sample image in the sample pair and image B is the second sample image in the sample pair. Inputting image A into an image encoder yields the implicit features of image A (also known as image embedding features, i.e., the sample image features in the above embodiments), which corresponds to step S3011 in the above embodiments. Figure 9 Image A is binarized to obtain a segmentation mask (i.e., the sample mask image in the above embodiment), that is, step S302 in the above embodiment. The white area represents the image area occupied by the preset type of object (i.e., a person's face) in image A, and the black area represents the other parts of image A besides the image area occupied by the preset type of object. The text prompt "an image of a young girl's face" is input into the frozen CLIP text encoder to obtain text embedding features (i.e., the sample text features in the above embodiment), that is, step S3031 in the above embodiment.

[0216] The obtained text embedding features, image embedding features, and segmentation mask are used as conditional embedding inputs to the denoiser. The text embedding features control the image content of the generated image, while the image embedding features and segmentation mask define the position and size of the image region occupied by a preset type of object in the generated image. During forward diffusion, the denoiser progressively adds noise (i.e., the preset noise in the above embodiment) to image B, i.e., step S3041 in the above embodiment. During backward diffusion, denoising is progressively performed to generate the image, i.e., steps S3051-S3053 in the above embodiment. Thus, an image can be generated through a diffusion process. This diffusion process uses the implicit features of image A as conditions, allowing the image generation model to learn a compact latent feature space representation. Instead of compressing all image information into this feature space representation, it focuses on the most descriptive content of the image, i.e., the position and size of the image region occupied by the preset type of object.

[0217] Based on the above processing, the pre-trained image encoder can be directly used for downstream face detection tasks. The implicit features obtained from an image containing a human face through the image encoder have the ability to initially perceive the position and size of the human face. Furthermore, using the trained image generation model, a new image containing a human face can be generated from an arbitrary reference image containing a human face and labeled with information indicating the position of the human face. The position of the human face in the generated image is consistent with the position of the human face in the given reference image.

[0218] Thus, the embodiments of this application can provide a method for pre-training and sample generation of human face detection based on diffusion networks. This method can expand the human face detection dataset while performing pre-training for human face detection, thus solving the problem of high cost of expanding the dataset and the problem of insufficient generalization performance of human face detection by learning general visual features.

[0219] Based on the same inventive concept, this application also provides an image generation method, see [link to relevant documentation]. Figure 10 , Figure 10 This is a flowchart illustrating an image generation method provided in an embodiment of this application. The method includes:

[0220] Step S1001: Obtain a reference image containing objects of a preset type, and a bounding box representing the image area occupied by the objects of the preset type;

[0221] Step S1002: Input the reference image, the preset description text, and the preset noisy image into the pre-trained image generation model to obtain an image to be used that conforms to the preset description text and contains objects of the preset type.

[0222] The image region occupied by the preset type of object in the image to be used has the same position and size as the image region occupied by the preset type of object in the reference image; the preset noise image has the same size as the reference image; and the image generation model is trained based on any of the image generation model training methods described above.

[0223] Based on the above processing, since the image generation model uses descriptive text during training, it can learn the features of the descriptive text, thus enabling the output image to match the descriptive text. Furthermore, for each sample pair, since the two sample images are of the same size, both contain objects of a preset type, and the bounding boxes in the two sample images have the same position and size, the image generation model can learn the features of the bounding box position and size in the image. This ensures that the position of the object in the output image matches the position of the object in the sample image, and the size of the image region occupied by the object in the output image matches the size of the region occupied by the object in the sample image.

[0224] Thus, by inputting a reference image, preset descriptive text, and preset noisy images into a pre-trained image generation model, a usable image can be obtained that conforms to the preset descriptive text and contains objects of a preset type. Furthermore, the position of the object in the input reference image is consistent with the position of the object in the obtained usable image, and the size of the image region occupied by the object in the reference image is also consistent with the size of the image region occupied by the object in the usable image. That is, based on the position and size of the bounding boxes in the input reference image, the position and size of the object in the obtained usable image can be determined, eliminating the need for manual determination and labeling of the object's position and size. This reduces labor costs and improves the efficiency of acquiring sample images.

[0225] In one embodiment, step S1002 includes performing the following steps using a pre-trained image generation model:

[0226] Step 1: Extract image features from the reference image to obtain the image features to be used;

[0227] Step 2: Binarize the reference image according to the position of the marked box in the reference image to obtain the mask image to be used;

[0228] Step 3: Extract text features from the preset description text to obtain the text features to be used;

[0229] Step 4: Using a diffusion network, the preset noise image is back-diffused with the mask image to be used, the features of the image to be used, and the features of the text to be used to obtain a target image that conforms to the preset description text and contains an object of the preset type.

[0230] In this embodiment of the application, the image generation model can be based on Figure 3 The image generation model is trained using the method shown. Step 1 can be referred to the relevant description of step S301 in the above embodiments, step 2 can be referred to the relevant description of step S302 in the above embodiments, step 3 can be referred to the relevant description of step S303 in the above embodiments, and step 4 can be referred to the relevant description of step S305 in the above embodiments.

[0231] In one embodiment, step 4 includes:

[0232] Step 41: The current fusion result to be used is fused with the mask image to be used through a diffusion network to obtain the current first fusion result to be used; wherein, the fusion result to be used when the first fusion is performed to obtain the first fusion result to be used is the preset noise image;

[0233] Step 42: Using the diffusion network based on the cross-attention mechanism, the current first fusion result to be utilized is processed using the image features to be utilized and the text features to be utilized to obtain the current processing result to be utilized;

[0234] Step 43: Denoise the current first fusion result to be utilized using the diffusion network to obtain a new fusion result to be utilized, and return to step 41 until the number of denoising operations reaches a specified number, so as to obtain an image to be utilized that conforms to the preset description text and contains the preset type of object.

[0235] In one embodiment, the image generation model further includes a text feature extraction layer and an image feature extraction layer, and the diffusion network includes an encoding layer, a decoding layer, a first multi-head attention layer, and a second multi-head attention layer;

[0236] Step 1 includes: Step 11: Input the reference image into the image feature extraction layer to obtain the image features to be used;

[0237] Step 3 includes: Step 31: Input the preset descriptive text into the text feature extraction layer to obtain the text features to be utilized;

[0238] Step 42 includes:

[0239] Step 421: Use the coding layer to encode the current first fusion result to be used, and obtain the coding result to be used;

[0240] Step 422: Using the first multi-head attention layer, process the obtained encoding result to be used and the image features to be used to obtain a first attention result to be used; and using the second multi-head attention layer, process the obtained encoding result to be used and the text features to be used to obtain a second attention result to be used; and fuse the first attention result to be used and the second attention result to be used to obtain a second fusion result to be used.

[0241] Step 423: Use the decoding layer to decode the obtained second fusion result to obtain the current processing result.

[0242] In this embodiment, step 11 can refer to the relevant description of step S3011 in the above embodiments, and step 31 can refer to the relevant description of step S3031 in the above embodiments. Steps 421-423 correspond to steps S3051-S3053 in the above embodiments, and can refer to the relevant descriptions of steps S3051-S3053 in the above embodiments, respectively.

[0243] In one embodiment, the diffusion network is implemented based on Unet, and a first multi-head attention layer and a second multi-head attention layer are provided between each encoding layer and its corresponding decoding layer; the query of the first multi-head attention layer and the second multi-head attention layer corresponding to each encoding layer is obtained based on the output data of the encoding layer, and the data obtained by performing different linear transformations on the image features to be utilized are respectively the key and value of the first multi-head attention layer; the data obtained by performing different linear transformations on the text features to be utilized are respectively the key and value of the second multi-head attention layer.

[0244] In one embodiment, the method further includes:

[0245] Using the image feature extraction layer, feature extraction is performed on the second image to be detected, which contains objects of the preset type, to obtain the features of the second image to be detected;

[0246] The features of the second image to be detected are input into the classification layer to obtain the image detection result of the second image to be detected.

[0247] In this embodiment, during the training of the image generation model, for each sample pair, since the two sample images are of the same size, both contain objects of a preset type, and the bounding boxes of the two sample images are in the same position and size, the image feature extraction layer of the image generation model can learn to extract the most significant common features between the two images in the sample pair. That is, both images contain objects of a preset type, the objects of the preset type are in the same position, and the size of the image region occupied by the objects is also the same. Therefore, the image feature extraction layer can learn to extract the features of the preset type of objects and the features of the position of the image region occupied by the objects in the image.

[0248] Furthermore, using the image feature extraction layer in the trained image generation model, features can be extracted from the second image to be detected that contains objects of the preset type. The extracted features (i.e., the features of the second image to be detected) can include: the features of the objects of the preset type in the second image to be detected, and the features of the location of the image regions occupied by the objects of the preset type in the second image to be detected. Accordingly, the features of the second image to be detected can be used for subsequent image detection of the second image to detect objects of the preset type contained in the second image to be detected. For example, by inputting the features of the second image to be detected into the classification layer, the probability of objects of the preset type in the second image to be detected belonging to each preset category can be obtained. Thus, the image feature extraction layer in the trained image generation model can also be applied to subsequent image detection tasks.

[0249] In the technical solution of this application, the operations of acquiring, storing, using, processing, transmitting, providing and disclosing user facial images are all carried out with the user's authorization.

[0250] Based on the same inventive concept, this application also provides an image generation model training device, see [link to relevant documentation]. Figure 11 , Figure 11 This is a schematic diagram of the structure of an image generation model training device provided in an embodiment of this application. The device includes:

[0251] The sample pair acquisition module 1101 is used to acquire multiple sample pairs; wherein each sample pair includes two sample images of the same size, both containing objects of a preset type; both sample images have a bounding box indicating the image area occupied by the object, and the bounding boxes of the two sample images are in the same position and size in their respective images.

[0252] The prediction image determination module 1102 is used to input the first sample image and the second sample image in each sample pair, as well as the sample description text used to describe the second sample image in the sample pair, into the image generation model of the initial structure to obtain the prediction image output by the image generation model of the initial structure.

[0253] The model parameter adjustment module 1103 is used to adjust the model parameters of the image generation model of the initial structure based on the difference between the predicted image obtained by the diffusion network in the image generation model of the initial structure and the second sample image in the sample pair when processing each sample pair, until the preset convergence condition is reached, and the trained image generation model is obtained.

[0254] Based on the image generation model training apparatus provided in this application embodiment, multiple sample pairs can be acquired, and the image generation model can be trained using two sample images and a descriptive text for one of the images in each sample pair. Since the descriptive text is used during the training process, the image generation model can learn the features of the descriptive text, thus enabling the output image to conform to the descriptive text. Furthermore, for each sample pair, since the two sample images are of the same size, both contain objects of a preset type, and the bounding boxes of the two sample images have the same position and size, the image generation model can learn the features of the bounding box position and size in the image. This ensures that the position of the object in the output image is consistent with the position of the object in the sample image, and the size of the image area occupied by the object in the output image is also consistent with the size of the area occupied by the object in the sample image. Thus, the image obtained by the image generation model conforms to the input descriptive text, and the position and size of the object in the image obtained by the image generation model can be determined based on the position and size of the bounding box in the input image, eliminating the need for manual determination and labeling of the object's position and size, reducing labor costs, and improving the efficiency of acquiring sample images.

[0255] In one embodiment, the predicted image determination module 1102 is specifically used to perform the following steps using an image generation model with an initial structure:

[0256] Image features are extracted from the first sample image in each sample pair to obtain sample image features;

[0257] Based on the position of the bounding box in the first sample image, the first sample image is binarized to obtain a sample mask image;

[0258] Text features are extracted from the sample description text used to describe the second sample image in the sample pair to obtain the sample text features;

[0259] The second sample image in the sample pair is forward diffused through the diffusion network to obtain the processing result of the first sample.

[0260] The first sample processing result is back-diffused using the diffusion network, the sample mask image, the sample image features, and the sample text features to obtain a predicted image.

[0261] In one embodiment, the second sample image in the sample pair is forward diffused through the diffusion network to obtain the first sample processing result, including:

[0262] A specified number of preset noises are generated through the diffusion network and added sequentially to the second sample image of the sample pair to obtain the first sample processing result;

[0263] The first sample processing result is back-diffused using the sample mask image, the sample image features, and the sample text features through the diffusion network to obtain a predicted image. This includes fusing the current sample fusion result with the sample mask image through the diffusion network to obtain the current first sample fusion result; wherein the sample fusion result used when obtaining the first sample fusion result for the first time is the first sample processing result.

[0264] The diffusion network, based on a cross-attention mechanism, processes the current first sample fusion result using the sample image features and the sample text features to obtain the current second sample processing result.

[0265] The diffusion network is used to denoise the current sample fusion result using the current second sample processing result to obtain a new sample fusion result. Then, the process returns to the step of fusing the current sample fusion result with the sample mask image through the diffusion network to obtain the current first sample fusion result. This process continues until the number of denoising operations reaches the specified number, and a predicted image is obtained.

[0266] The model parameter adjustment module 1103 is specifically used to calculate the sub-loss value corresponding to each second sample processing result based on the difference between the second sample processing result and the corresponding preset noise; wherein, the i-th second sample processing result obtained in the order of denoising corresponds to the (n+1)-i-th added preset noise, and n represents a specified number; calculate the loss value based on the sub-loss value corresponding to each second sample processing result; and adjust the model parameters of the image generation model of the initial structure according to the calculated loss value until the preset convergence condition is reached to obtain the trained image generation model.

[0267] In one embodiment, the image generation model of the initial structure further includes: a text feature extraction layer and an image feature extraction layer, and the diffusion network includes: an encoding layer, a decoding layer, a first multi-head attention layer and a second multi-head attention layer;

[0268] Image feature extraction is performed on the first sample image in each sample pair to obtain sample image features, including: inputting the first sample image in each sample pair into the image feature extraction layer to obtain sample image features;

[0269] The text feature extraction of the sample description text used to describe the second sample image in the sample pair is performed to obtain sample text features, including: inputting the sample description text used to describe the second sample image in the sample pair into the text feature extraction layer to obtain sample text features;

[0270] The diffusion network, based on a cross-attention mechanism, processes the current first sample fusion result using the sample image features and the sample text features to obtain the current second sample processing result, including:

[0271] The current first sample fusion result is encoded using the encoding layer to obtain the sample encoding result;

[0272] The first multi-head attention layer is used to process the obtained sample encoding result and the sample image features to obtain a first sample attention result. The second multi-head attention layer is used to process the obtained sample encoding result and the sample text features to obtain a second sample attention result. The first sample attention result and the second sample attention result are then fused to obtain a second sample fusion result.

[0273] The obtained second sample fusion result is decoded using the decoding layer to obtain the current second sample processing result.

[0274] In one embodiment, the text feature extraction layer is the part of a pre-trained image-text pre-training model used to extract text features, and the parameters of the text feature extraction layer remain unchanged during the training of the image generation model with the initial structure.

[0275] In one embodiment, the diffusion network is implemented based on Unet, and a first multi-head attention layer and a second multi-head attention layer are set between each encoding layer and its corresponding decoding layer; the query of the first and second multi-head attention layers corresponding to each encoding layer is obtained based on the output data of that encoding layer; the data obtained by performing different linear transformations on the sample image features are respectively the key and value of the first multi-head attention layer; the data obtained by performing different linear transformations on the sample text features are respectively the key and value of the second multi-head attention layer.

[0276] In one embodiment, the apparatus further includes:

[0277] The first image feature extraction module is used to extract features from the first image to be detected containing the preset type of object using the image feature extraction layer in the trained image generation model, thereby obtaining the first image features; wherein, the first image features are used to perform image detection on the first image to be detected.

[0278] In one embodiment, a sample pair is obtained based on the following steps:

[0279] Select any two images from the pre-acquired images that contain objects of the preset type and have the labeled boxes of the objects;

[0280] By performing image transformation on two selected images so that the two transformed images are the same size and the position and size of the bounding boxes in each image are the same, a sample pair is obtained.

[0281] Based on the same inventive concept, this application also provides an image generation apparatus, see [link to relevant documentation]. Figure 12 , Figure 12 This is a schematic diagram of an image generation apparatus provided in an embodiment of this application. The apparatus includes:

[0282] The reference image acquisition module 1201 is used to acquire a reference image containing objects of a preset type and a bounding box representing the image area occupied by the objects of the preset type.

[0283] The image generation module 1202 is used to input the reference image, preset descriptive text, and preset noise image into a pre-trained image generation model to obtain an image to be used that conforms to the preset descriptive text and contains an object of the preset type; wherein, the image region occupied by the object of the preset type in the image to be used is the same in position and size as the image region occupied by the object of the preset type in the reference image; the preset noise image is the same size as the reference image; and the image generation model is trained based on any of the image generation model training methods described above.

[0284] Based on the image generation apparatus provided in this application embodiment, since descriptive text is used during the training process of the image generation model, the image generation model can learn the features of the descriptive text, thus enabling the output image to conform to the descriptive text. Furthermore, for each sample pair, since the two sample images are of the same size, both contain objects of a preset type, and the bounding boxes of the two sample images have the same position and size, the image generation model can learn the features of the bounding box position and size in the image. This allows the position of the object in the output image to be consistent with the position of the object in the sample image, and the size of the image area occupied by the object in the output image is also consistent with the size of the area occupied by the object in the sample image.

[0285] Thus, by inputting a reference image, preset descriptive text, and preset noisy images into a pre-trained image generation model, a usable image can be obtained that conforms to the preset descriptive text and contains objects of a preset type. Furthermore, the position of the object in the input reference image is consistent with the position of the object in the obtained usable image, and the size of the image region occupied by the object in the reference image is also consistent with the size of the image region occupied by the object in the usable image. That is, based on the position and size of the bounding boxes in the input reference image, the position and size of the object in the obtained usable image can be determined, eliminating the need for manual determination and labeling of the object's position and size. This reduces labor costs and improves the efficiency of acquiring sample images.

[0286] In one embodiment, the image generation module 1202 is specifically used to perform the following steps using a pre-trained image generation model:

[0287] Image features are extracted from the reference image to obtain the image features to be used;

[0288] The reference image is binarized according to the position of the marked box in the reference image to obtain the mask image to be used;

[0289] Text features are extracted from the preset descriptive text to obtain the text features to be utilized.

[0290] By using a diffusion network, the preset noisy image is back-diffused using the mask image to be used, the features of the image to be used, and the features of the text to be used, to obtain a usable image that conforms to the preset descriptive text and contains an object of the preset type.

[0291] In one embodiment, the step of performing back-diffusion processing on the preset noisy image using a diffusion network, utilizing the mask image to be used, the features of the image to be used, and the text features to be used, to obtain a usable image that conforms to the preset descriptive text and contains an object of the preset type, includes:

[0292] The current fusion result to be utilized is fused with the mask image to be utilized through a diffusion network to obtain the current first fusion result to be utilized; wherein, the fusion result to be utilized used when the first fusion is performed to obtain the first fusion result to be utilized is the preset noise image;

[0293] The diffusion network, based on the cross-attention mechanism, processes the current first fusion result to be utilized using the image features and text features to be utilized, thereby obtaining the current processing result to be utilized.

[0294] The diffusion network is used to denoise the current first fusion result to be utilized using the current processing result to obtain a new fusion result to be utilized. Then, the process returns to the step of fusing the current fusion result to be utilized with the mask image to be utilized using the diffusion network to obtain the current first fusion result to be utilized. This process continues until the number of denoising operations reaches a specified number, resulting in a fusion image to be utilized that conforms to the preset description text and contains objects of the preset type.

[0295] In one embodiment, the image generation model further includes a text feature extraction layer and an image feature extraction layer, and the diffusion network includes an encoding layer, a decoding layer, a first multi-head attention layer, and a second multi-head attention layer;

[0296] Image feature extraction is performed on the reference image to obtain the image features to be used, including:

[0297] The reference image is input into the image feature extraction layer to obtain the image features to be used;

[0298] Text feature extraction is performed on the preset descriptive text to obtain the text features to be utilized, including:

[0299] Input the preset descriptive text into the text feature extraction layer to obtain the text features to be utilized;

[0300] The diffusion network, based on a cross-attention mechanism, processes the current first fusion result using the image features and text features to be utilized, to obtain the current processing result, including:

[0301] The current first fusion result to be utilized is encoded using the encoding layer to obtain the encoded result to be utilized.

[0302] The first multi-head attention layer is used to process the obtained encoding result to be used and the image features to be used to obtain a first attention result to be used. The second multi-head attention layer is used to process the obtained encoding result to be used and the text features to be used to obtain a second attention result to be used. The first attention result to be used and the second attention result to be used are fused to obtain a second fused result to be used.

[0303] The second fusion result to be utilized is decoded using the decoding layer to obtain the current processing result to be utilized.

[0304] In one embodiment, the diffusion network is implemented based on Unet, and a first multi-head attention layer and a second multi-head attention layer are provided between each encoding layer and its corresponding decoding layer; the query of the first multi-head attention layer and the second multi-head attention layer corresponding to each encoding layer is obtained based on the output data of the encoding layer, and the data obtained by performing different linear transformations on the image features to be utilized are respectively the key and value of the first multi-head attention layer; the data obtained by performing different linear transformations on the text features to be utilized are respectively the key and value of the second multi-head attention layer.

[0305] In one embodiment, the apparatus further includes:

[0306] The second image feature extraction module is used to extract features from the second image to be detected containing the preset type of object using the image feature extraction layer, so as to obtain the features of the second image to be detected.

[0307] The image detection result determination module is used to input the features of the second image to be detected into the classification layer to obtain the image detection result of the second image to be detected.

[0308] In one embodiment, the step of denoising the obtained preset noise image using the noise obtained through the diffusion network to obtain a usable image that conforms to the preset description text and contains an object of the preset type includes:

[0309] The obtained preset noise image is denoised using the obtained noise to be utilized through the diffusion network to obtain a new preset noise image. Then, the process of fusing the current preset noise image with the mask image to be utilized to obtain a first fusion result is returned until the number of times the obtained preset noise image is denoised reaches the specified number, so as to obtain a usable image that conforms to the preset description text and contains an object of the preset type.

[0310] This application also provides an electronic device, such as... Figure 13 As shown, it includes:

[0311] Memory 1301 is used to store computer programs;

[0312] The processor 1302 is used to execute the program stored in the memory 1301 to implement any of the above-described image generation model training methods or image generation methods.

[0313] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 1302, the communication interface, and the memory 1301 communicating with each other via the communication bus.

[0314] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0315] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0316] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0317] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0318] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described image generation model training methods or image generation methods.

[0319] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the image generation model training methods or image generation methods described above.

[0320] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0321] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0322] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of the image generation method, apparatus, electronic device, storage medium, and program product are basically similar to the embodiments of the image generation model training method, so the description is relatively simple; relevant parts can be referred to in the description of the image generation model training method embodiments.

[0323] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for training an image generation model, characterized in that, The method includes: Multiple sample pairs are obtained; wherein each sample pair includes two sample images of the same size, both containing objects of a preset type; both sample images have bounding boxes representing the image area occupied by the object, and the bounding boxes of the two sample images are in the same position and size in their respective images. Input the first and second sample images of each sample pair, along with the sample description text used to describe the second sample image in the sample pair, into the image generation model of the initial structure to obtain the predicted image output by the image generation model of the initial structure. Based on the difference between the predicted image obtained by the diffusion network in the initial image generation model and the second sample image in the sample pair when processing each sample pair, the model parameters of the initial image generation model are adjusted until the preset convergence condition is reached, and the trained image generation model is obtained.

2. The method according to claim 1, characterized in that, The step of inputting the first and second sample images of each sample pair, along with the sample description text describing the second sample image in the sample pair, into the image generation model of the initial structure to obtain the predicted image output by the image generation model of the initial structure includes: The following steps are performed using an image generation model based on the initial structure: Image features are extracted from the first sample image in each sample pair to obtain sample image features; Based on the position of the bounding box in the first sample image, the first sample image is binarized to obtain a sample mask image; Text features are extracted from the sample description text used to describe the second sample image in the sample pair to obtain the sample text features; The second sample image in the sample pair is forward diffused through the diffusion network to obtain the processing result of the first sample. The first sample processing result is back-diffused using the diffusion network, the sample mask image, the sample image features, and the sample text features to obtain a predicted image.

3. The method according to claim 2, characterized in that, The second sample image in the sample pair is forward diffused through the diffusion network to obtain the first sample processing result, including: A specified number of preset noises are generated through the diffusion network and added sequentially to the second sample image of the sample pair to obtain the first sample processing result; The first sample processing result is back-diffused using the diffusion network, the sample mask image, the sample image features, and the sample text features to obtain a predicted image, including: The current sample fusion result is fused with the sample mask image through the diffusion network to obtain the current first sample fusion result; wherein, the sample fusion result used when the first fusion is performed to obtain the first sample fusion result is the first sample processing result; The diffusion network, based on a cross-attention mechanism, processes the current first sample fusion result using the sample image features and the sample text features to obtain the current second sample processing result. The diffusion network is used to denoise the current sample fusion result using the current second sample processing result to obtain a new sample fusion result. Then, the process returns to the step of fusing the current sample fusion result with the sample mask image through the diffusion network to obtain the current first sample fusion result. This process continues until the number of denoising operations reaches the specified number, and a predicted image is obtained. The step of adjusting the model parameters of the initial image generation model based on the difference between the predicted image obtained by the diffusion network in the initial image generation model and the second sample image in the sample pair when processing each sample pair, until a preset convergence condition is reached, to obtain the trained image generation model, includes: For each second sample processing result, a sub-loss value corresponding to the second sample processing result is calculated based on the difference between the second sample processing result and the corresponding preset noise; wherein, the i-th second sample processing result obtained in the order of denoising corresponds to the (n+1)-i-th added preset noise, and n represents the specified number; Calculate the loss value based on the sub-loss value corresponding to the processing result of each second sample; Based on the calculated loss value, the model parameters of the initial image generation model are adjusted until the preset convergence condition is met, thus obtaining the trained image generation model.

4. The method according to claim 3, characterized in that, The initial structure of the image generation model also includes: a text feature extraction layer and an image feature extraction layer, and the diffusion network includes: an encoding layer, a decoding layer, a first multi-head attention layer and a second multi-head attention layer; For each sample pair, image features are extracted from the first sample image to obtain sample image features, including: The first sample image in each sample pair is input into the image feature extraction layer to obtain the sample image features; Text feature extraction is performed on the sample description text used to describe the second sample image in the sample pair to obtain sample text features, including: The sample description text used to describe the second sample image in the sample pair is input into the text feature extraction layer to obtain the sample text features; The diffusion network, based on a cross-attention mechanism, processes the current first sample fusion result using the sample image features and the sample text features to obtain the current second sample processing result, including: The current first sample fusion result is encoded using the encoding layer to obtain the sample encoding result; The first multi-head attention layer is used to process the obtained sample encoding result and the sample image features to obtain a first sample attention result. The second multi-head attention layer is used to process the obtained sample encoding result and the sample text features to obtain a second sample attention result. The first sample attention result and the second sample attention result are then fused to obtain a second sample fusion result. The obtained second sample fusion result is decoded using the decoding layer to obtain the current second sample processing result.

5. The method according to claim 4, characterized in that, The text feature extraction layer is the part of the pre-trained image-text pre-training model used to extract text features, and the parameters of the text feature extraction layer remain unchanged during the training of the image generation model with the initial structure.

6. The method according to claim 4, characterized in that, The diffusion network is implemented based on Unet, and a first multi-head attention layer and a second multi-head attention layer are set between each encoding layer and its corresponding decoding layer; the query of the first and second multi-head attention layers corresponding to each encoding layer is obtained based on the output data of that encoding layer; the data obtained by performing different linear transformations on the sample image features are respectively the key and value of the first multi-head attention layer; the data obtained by performing different linear transformations on the sample text features are respectively the key and value of the second multi-head attention layer. And / or, The method further includes: Using the image feature extraction layer in the trained image generation model, feature extraction is performed on the first image to be detected containing the object of the preset type to obtain the first image to be detected features; wherein, the first image to be detected features are used to perform image detection on the first image to be detected.

7. The method according to claim 1, characterized in that, A sample pair is obtained based on the following steps: Select any two images from the pre-acquired images that contain objects of the preset type and have the labeled boxes of the objects; By performing image transformation on two selected images so that the two transformed images are the same size and the position and size of the bounding boxes in each image are the same, a sample pair is obtained.

8. An image generation method, characterized in that, The method includes: Obtain a reference image containing objects of a preset type, and a bounding box representing the image area occupied by the objects of the preset type; The reference image, preset descriptive text, and preset noise image are input into a pre-trained image generation model to obtain a usable image that conforms to the preset descriptive text and contains an object of the preset type; wherein, the image region occupied by the object of the preset type in the usable image has the same position and size as the image region occupied by the object of the preset type in the reference image; the preset noise image has the same size as the reference image; the image generation model is trained based on the method described in any one of claims 1-7.

9. The method according to claim 8, characterized in that, The reference image, preset descriptive text, and preset noisy image are input into a pre-trained image generation model to obtain a usable image that conforms to the preset descriptive text and contains an object of the preset type, including: Perform the following steps using a pre-trained image generation model: Image features are extracted from the reference image to obtain the image features to be used; The reference image is binarized according to the position of the marked box in the reference image to obtain the mask image to be used; Text features are extracted from the preset descriptive text to obtain the text features to be utilized. By using a diffusion network, the preset noisy image is back-diffused using the mask image to be used, the features of the image to be used, and the features of the text to be used, to obtain a usable image that conforms to the preset descriptive text and contains an object of the preset type.

10. The method according to claim 9, characterized in that, The step of using a diffusion network to perform back-diffusion processing on the preset noisy image using the mask image to be used, the features of the image to be used, and the features of the text to be used, to obtain a usable image that conforms to the preset description text and contains an object of the preset type, includes: The current fusion result to be utilized is fused with the mask image to be utilized through a diffusion network to obtain the current first fusion result to be utilized; wherein, the fusion result to be utilized used when the first fusion is performed to obtain the first fusion result to be utilized is the preset noise image; The diffusion network, based on the cross-attention mechanism, processes the current first fusion result to be utilized using the image features and text features to be utilized, thereby obtaining the current processing result to be utilized. The diffusion network is used to denoise the current first fusion result to be utilized using the current processing result to obtain a new fusion result to be utilized. Then, the process returns to the step of fusing the current fusion result to be utilized with the mask image to be utilized using the diffusion network to obtain the current first fusion result to be utilized. This process continues until the number of denoising operations reaches a specified number, resulting in a fusion image to be utilized that conforms to the preset description text and contains objects of the preset type.

11. The method according to claim 10, characterized in that, The image generation model further includes a text feature extraction layer and an image feature extraction layer, and the diffusion network includes an encoding layer, a decoding layer, a first multi-head attention layer, and a second multi-head attention layer. Image feature extraction is performed on the reference image to obtain the image features to be used, including: The reference image is input into the image feature extraction layer to obtain the image features to be used; Text feature extraction is performed on the preset descriptive text to obtain the text features to be utilized, including: Input the preset descriptive text into the text feature extraction layer to obtain the text features to be utilized; The diffusion network, based on a cross-attention mechanism, processes the current first fusion result using the image features and text features to be utilized, to obtain the current processing result, including: The current first fusion result to be utilized is encoded using the encoding layer to obtain the encoded result to be utilized. The first multi-head attention layer is used to process the obtained encoding result to be used and the image features to be used to obtain a first attention result to be used. The second multi-head attention layer is used to process the obtained encoding result to be used and the text features to be used to obtain a second attention result to be used. The first attention result to be used and the second attention result to be used are fused to obtain a second fused result to be used. The second fusion result to be utilized is decoded using the decoding layer to obtain the current processing result to be utilized.

12. The method according to claim 11, characterized in that, The diffusion network is implemented based on Unet, and a first multi-head attention layer and a second multi-head attention layer are set between each encoding layer and its corresponding decoding layer; the query of the first multi-head attention layer and the second multi-head attention layer corresponding to each encoding layer is obtained based on the output data of that encoding layer, and the data obtained by performing different linear transformations on the image features to be utilized are respectively the key and value of the first multi-head attention layer; the data obtained by performing different linear transformations on the text features to be utilized are respectively the key and value of the second multi-head attention layer. And / or, The method further includes: Using the image feature extraction layer, feature extraction is performed on the second image to be detected, which contains objects of the preset type, to obtain the features of the second image to be detected; The features of the second image to be detected are input into the classification layer to obtain the image detection result of the second image to be detected.

13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method according to any one of claims 1-7 or any one of claims 8-12.

Images