Image generation, model training method, device, equipment and storage medium

Abstract

The present disclosure relates to an image generation, model training method, device, equipment and storage medium. The present disclosure generates a target image through multiple image generation elements, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generation elements one by one, and the similarity between the corresponding target elements and the image generation elements is greater than or equal to a preset value. Since the number of image generation elements is large, and the combination of different image generation elements is also flexible and diverse, a large number of different target images can be generated by combining different multiple image generation elements, which greatly improves the diversity of target images.

Description

Technical Field

[0001] The present disclosure relates to the field of information technology, and in particular to an image generation, model training method, device, equipment and storage medium. Background Art

[0002] As images are widely used, image generation technology is also developing continuously. For example, in the current AI generated content (AIGC) solution, the image generation model can automatically generate images.

[0003] However, the images generated by current image generation models are not diverse enough, resulting in the images generated by image generation models not being able to be widely used. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an image generation, model training method, device, equipment and storage medium, which can generate a large number of different target images by combining different multiple image generation elements, thereby greatly improving the diversity of target images.

[0005] In a first aspect, an embodiment of the present disclosure provides an image generation method, including:

[0006] Get multiple image generation elements;

[0007] A target image is generated based on the multiple image generating elements, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generating elements one-to-one, and the similarity between the corresponding target elements and the image generating elements is greater than or equal to a preset value.

[0008] In a second aspect, the present disclosure provides an image generation method, including:

[0009] receiving a plurality of image generation elements sent by a terminal;

[0010] Generate a target image according to the multiple image generating elements, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generating elements one by one, and the similarity between the corresponding target elements and the image generating elements is greater than or equal to a preset value;

[0011] The target image is fed back to the terminal.

[0012] In a third aspect, an embodiment of the present disclosure provides an image generation method, the method comprising:

[0013] In response to the plurality of image generating elements selected by the user, a target image is generated, so that the plurality of image elements of the target image include a plurality of target elements, the plurality of target elements correspond to the plurality of image generating elements one by one, and the similarity between the corresponding target elements and the image generating elements is greater than or equal to a preset value;

[0014] The target image is displayed.

[0015] In a fourth aspect, an embodiment of the present disclosure provides an image generation method, the method comprising:

[0016] Decomposing at least one sample image into a plurality of image elements respectively;

[0017] constructing a database according to a plurality of image elements respectively corresponding to the at least one sample image and identifiers of the image elements;

[0018] receiving multiple identifiers sent by a terminal;

[0019] According to the multiple identifiers, acquiring multiple image elements corresponding to the multiple identifiers from the database;

[0020] A target image is generated based on multiple image elements corresponding to the multiple identifiers, so that the multiple image elements of the target image include multiple target elements, the multiple image elements corresponding to the multiple identifiers correspond to the multiple target elements one-to-one, and the similarity between the corresponding target elements and the image elements is greater than or equal to a preset value.

[0021] In a fifth aspect, an embodiment of the present disclosure provides a model training method, the method comprising:

[0022] Acquire a sample image and a noisy image corresponding to the sample image;

[0023] Decompose the sample image into multiple image elements;

[0024] Inputting the noisy image and at least part of the multiple image elements into a machine learning model to be trained, so that the machine learning model to be trained performs denoising processing on the noisy image according to the at least part of the image elements to obtain a denoised image;

[0025] According to the sample image and the denoised image, the machine learning model to be trained is trained to obtain a trained machine learning model, and the trained machine learning model is used to implement the image generation method as described above.

[0026] In a sixth aspect, an embodiment of the present disclosure provides an image generating device, including:

[0027] An acquisition module, used for acquiring multiple image generation elements;

[0028] A generation module is used to generate a target image based on the multiple image generation elements, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generation elements one by one, and the similarity between the corresponding target elements and the image generation elements is greater than or equal to a preset value.

[0029] In a seventh aspect, an embodiment of the present disclosure provides an electronic device, including:

[0030] Memory;

[0031] Processor; and

[0032] Computer programs;

[0033] The computer program is stored in the memory and is configured to be executed by the processor to implement the methods described in the first to fifth aspects.

[0034] In an eighth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the methods described in the first to fifth aspects.

[0035] The image generation, model training method, device, equipment and storage medium provided by the embodiments of the present disclosure generate a target image through multiple image generation elements, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generation elements one by one, and the similarity between the corresponding target elements and the image generation elements is greater than or equal to a preset value. Since the number of image generation elements is large and the combination of different image generation elements is flexible and diverse, a large number of different target images can be generated by combining different multiple image generation elements, which greatly improves the diversity of target images. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0038] Figure 1 A flow chart of a model training method provided in an embodiment of the present disclosure;

[0039] Figure 2 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0040] Figure 3 A schematic diagram of an image element provided for an embodiment of the present disclosure;

[0041] Figure 4 A flow chart of an image generation method provided by an embodiment of the present disclosure;

[0042] Figure 5 An iterative denoising flow chart provided for another embodiment of the present disclosure;

[0043] Figure 6 A schematic diagram of a combination of image elements provided by another embodiment of the present disclosure;

[0044] Figure 7 A schematic diagram of introducing a global vector in UNet provided by another embodiment of the present disclosure;

[0045] Figure 8 A schematic diagram of the processing process inside UNet provided by another embodiment of the present disclosure;

[0046] Fig. 9 A schematic diagram of introducing local vectors into UNet provided by another embodiment of the present disclosure;

[0047] Fig.10 A schematic diagram of image generation conditions and image correspondence provided in an embodiment of the present disclosure;

[0048] Fig.11 A schematic diagram of a denoised image and a noise-added image provided by another embodiment of the present disclosure;

[0049] Fig.12 A flow chart of an image generating method provided by another embodiment of the present disclosure;

[0050] Fig.13 A schematic diagram of an application scenario provided by another embodiment of the present disclosure;

[0051] Fig.14 A flow chart of an image generating method provided by another embodiment of the present disclosure;

[0052] Fig.15 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0053] Fig.16 A flow chart of an image generating method provided by another embodiment of the present disclosure;

[0054] Fig.17 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0055] Fig.18 A schematic diagram of the structure of an image generating device provided by an embodiment of the present disclosure;

[0056] Fig.19 A schematic diagram of the structure of a model training device provided in an embodiment of the present disclosure;

[0057] Fig. 20 A schematic diagram of the structure of an electronic device embodiment provided by the present disclosure.

[0058] Reference numerals:

[0059] 32- Figure 3 Depth map in 33- Figure 3 Segmentation diagram 36- Figure 3 The mask image in

[0060] 35- Figure 3 Grayscale image 61- Figure 6 Image in (large image) 63- Figure 6 Image in (large image)

[0061] 65- Figure 6 Image in (large image) 67- Figure 6 Image in (large image) DETAILED DESCRIPTION

[0062] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0063] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0064] It should be noted that the multiple image generation elements (including but not limited to multiple image generation elements selected by the user, etc.) and target images (including but not limited to images generated by electronic devices, images expected by users, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0065] In addition, the image generation method or model training method provided in this application will involve the following terminology explanations, the details of which are as follows:

[0066] Generative models: methods that can model data distribution and generate images through sampling.

[0067] Diffusion model: A data generation model that generates data with a specified distribution from pure noise through iterative denoising.

[0068] Normally, an image generation model can automatically generate images. However, the images generated by the current image generation model are not diverse enough, resulting in the images generated by the image generation model not being widely used. In response to this problem, the embodiments of the present disclosure provide a model training method and an image generation method, wherein the machine learning model trained by the model training method can be applied to the image generation process, that is, the image generation method can be implemented by the trained machine learning model. Specifically, the model training method introduces the training process of the machine learning model, and the image generation method introduces the use process or reasoning process of the machine learning model. Normally, the machine learning model needs to be trained first, and after the training is completed, the trained machine learning model is used to generate images. Therefore, the model training method and the image generation method are introduced in turn in conjunction with specific embodiments below.

[0069] Figure 1 A flow chart of the model training method provided in an embodiment of the present disclosure. Figure 1 A flow chart of a model training method provided for an embodiment of the present disclosure. The method can be executed by a model training device, which can be implemented in software and / or hardware, and the device can be configured in an electronic device, such as a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc. The server can specifically be a server in the cloud, and the model training method can be executed by the cloud. Several computing nodes (cloud servers) can be deployed in the cloud, and each computing node has computing, storage and other processing resources. In the cloud, multiple computing nodes can be organized to provide a certain service. Of course, a computing node can also provide one or more services. The way the cloud provides the service can be to provide a service interface to the outside, and the user calls the service interface to use the corresponding service. The service interface includes a software development kit (Software Development Kit, referred to as SDK), an application programming interface (Application Programming Interface, referred to as API), and the like.

[0070] Specifically, the model training method described in this embodiment can be applied to Figure 2 The application scenario shown in Figure 1. Figure 2As shown, the application scenario includes a terminal 21 and a server 22, wherein the model training method can be executed by the server 22. After the server 22 trains a machine learning model through the model training method, the server 22 can retain the machine learning model locally or deploy the machine learning model on the terminal 21 or other servers. Figure 2 This method is described in detail. Figure 1 As shown, the specific steps of this method are as follows:

[0071] S101: Acquire a sample image and a noisy image corresponding to the sample image.

[0072] For example, a large number of sample images are stored locally in the server 22 or in a database corresponding to the server 22. This embodiment does not limit the source of the sample images. For example, the sample images may be images captured by the server 22 from the network, or images received by the server 22 from the terminal 21 or other servers. When the sample images come from the terminal 21, the sample images may be images captured by the terminal 21, or images pre-stored locally in the terminal 21, or images modified or adjusted by the user of the terminal 21 on the images captured by the terminal 21 or the images pre-stored locally in the terminal 21. The specific modification method or adjustment method may be some conventional image processing methods, such as image cropping, brightness adjustment, image sharpening, etc.

[0073] It can be understood that the process of training the machine learning model is a continuous iterative process. Assuming that in this embodiment, one training of the machine learning model is recorded as one iteration, when the number of iterations is greater than or equal to the preset number of times, or when the parameters of the machine learning model are stabilized and converged after iteration, the iteration can be stopped, that is, the training is completed. Specifically, in this embodiment, one iteration process requires a pair of samples, and the pair of samples includes a sample image and a noisy image corresponding to the sample image. For example, the server 22 can randomly extract a sample image from a local or corresponding database, and further, add noise to the sample image to obtain a noisy image corresponding to the sample image. This embodiment does not limit the specific noise adding method. For example, salt and pepper noise can be added to the sample image, that is, the intensity value of at least part of the pixel points in the sample image is modified to a black point or a white point, the intensity value of the black point is 0, and the intensity value of the white point is 255. For example, Gaussian noise, Poisson noise, speckle noise, etc. can also be added to the sample image. The specific noise adding principle will not be repeated here.

[0074] S102: Decompose the sample image into multiple image elements.

[0075] Specifically, the server 22 may decompose the sample image into multiple image elements. For example, the server 22 may use any open source pre-trained image understanding model to decompose the sample image into multiple different image elements. Figure 3 As shown, the sample image 30 is disassembled into 8 different image elements, which are the feature vector 31, depth map 32, segmentation map 33, sketch 34, grayscale map 35, masking 36, palette 37 and text description 38 of the sample image 30. Among them, the depth map and the segmentation map can be combined to be recorded as the shape of the sample image. The feature vector of the sample image includes the semantics of the sample image and the style of the sample image, that is, the feature vector can not only represent the semantics, but also the style. Semantics is the main content of a picture that is not related to the details, such as the image including a dog and two cats belongs to semantics. In addition, the feature vector, sketch, segmentation map and depth map of the sample image can be combined to be recorded as the content of the sample image. In this embodiment, each image element can also be represented by a vector.

[0076] In addition, image elements are not limited to the eight different types of image elements mentioned above. Specifically, any information that can be extracted from a sample image in a decoupled form and describes a certain aspect of the sample image can be used as an image element. In addition, in addition to the eight types of image elements mentioned above, it can also include statistics (such as histograms, color tables, gradient histograms, etc.), low-level detail features (such as edges, corners, etc.), and high-level semantic features (such as object types, scene types, relationships, attributes, etc.).

[0077] It is understandable that the number of image elements of a sample image is not limited to 8, and may be more or less, for example. Figure 3 The sample image 30 and the eight different image elements shown are only schematic illustrations and are not intended to be limiting. Figure 3 The image shown is a picture in which the face of the person in the sample image 30 is removed, masked or blurred, and the image may also be a picture in which other areas or other parts in the sample image 30 are removed, masked or blurred. In addition, the names of the image elements may not be fixed. For example, a sketch may also be called an outline, a sketch, a quick sketch, etc., and a color scheme may also be called a color scheme table.

[0078] S103. Input the noisy image and at least part of the multiple image elements into a machine learning model to be trained, so that the machine learning model to be trained performs denoising on the noisy image according to at least part of the image elements to obtain a denoised image.

[0079] Specifically, the machine learning model to be trained in this embodiment can be recorded as a combined generation model. In some embodiments, the combined generation model includes an image disassembly part and an image combination part. The image disassembly part can decompose the sample image into multiple image elements. For example, the image disassembly part can be implemented by the pre-trained image understanding model as described above. The image combination part can recombine multiple image elements into an image. Specifically, the image combination part can be implemented by a diffusion model, and the diffusion model can specifically be a multi-conditional diffusion model, that is, the input of the diffusion model includes multiple conditions, and the multiple conditions can specifically be multiple image elements. Specifically, the diffusion model is a generation model that slowly converts a noisy image into a generated image through an iterative denoising process, so that the generated image meets the requirements of the multiple conditions. In this case, if the pre-trained image understanding model can be a pre-trained model, then in the embodiment of the present disclosure, the diffusion model needs to be trained.

[0080] Alternatively, in some other embodiments, the combined generative model includes an image combination part, and the image disassembly part can be used as a previous part of the combined generative model. In this case, the combined generative model can be a diffusion model, and therefore, the diffusion model needs to be trained.

[0081] In addition, in the above two cases, the input or part of the input of the image combination part is derived from the image elements output by the image disassembly part. For example, the input of the image combination part includes the image elements output by the image disassembly part. Alternatively, the input of the image combination part includes the image elements after the image elements output by the image disassembly part are modified or adjusted.

[0082] In addition, the combined generative model can be implemented based on a variety of models. That is to say, the combined generative model can be implemented not only based on the diffusion model, but also can be implemented using other generative models such as the generative adversarial network (GAN), the variational autoencoder model (VAE), the flow model (Flow-based models), and the autoregressive model. The embodiments of the present disclosure do not limit the generative model paradigm based on which the specific implementation is based.

[0083] This embodiment takes the diffusion model as an example to introduce the training process of the diffusion model. For example, after the sample image is decomposed into 8 different image elements, some image elements can be selected from the 8 different image elements. The specific selection mechanism can be random extraction, user selection, sequential extraction, etc. The number of selected image elements is not limited here, for example, it can be 4. Further, the 4 image elements and the noisy image corresponding to the sample image are input into the diffusion model, so that the diffusion model iteratively denoises the noisy image according to the 4 image elements to obtain a denoised image, and the denoised image meets the requirements of the 4 image elements.

[0084] Alternatively, in other embodiments, the eight different image elements may be selected. Further, the eight different image elements and the noisy image corresponding to the sample image are input into the diffusion model, so that the diffusion model iteratively denoises the noisy image according to the eight different image elements to obtain a denoised image, and the denoised image meets the requirements of the eight image elements.

[0085] S104: Training the machine learning model to be trained according to the sample image and the denoised image.

[0086] In an iterative training process, since the output of the diffusion model, i.e., the denoised image as described above, may not be accurate enough, there is a certain difference between the denoised image and the original image, i.e., the sample image. Therefore, the parameters of the diffusion model can be adjusted according to the difference. So that the adjusted parameters participate in the next iterative training process, and the parameters of the diffusion model are adjusted again in the next iterative training process. That is to say, in the continuous iterative training process, the parameters of the diffusion model will be continuously adjusted until the number of iterations is greater than or equal to the preset number of times, or when the parameters of the diffusion model tend to be stable and converge, the iteration can be stopped, i.e., the training of the diffusion model is completed, and a trained diffusion model is obtained.

[0087] Specifically, the training process of the diffusion model includes a forward process and a reverse process, wherein in the forward process, the diffusion model can perform multiple noise addition processes on the input sample image. Exemplarily, the diffusion model can add Gaussian noise to the sample image multiple times and step by step based on pre-set parameters. It can also be considered that the diffusion model includes multiple diffusion noise addition layers, the first diffusion noise addition layer takes the sample image as input and performs noise addition process on it; the subsequent diffusion noise addition layer takes the noised image output by the previous diffusion noise addition layer as input, performs noise addition process again in the current diffusion noise addition layer, and outputs the result of this noise addition process as the input of the next diffusion noise addition layer. After continuous and sequential noise addition process, the noise image output by the last diffusion noise addition layer of the forward process will be used as the input of the reverse process for the subsequent image denoising process. Among them, the noise image output by the last diffusion noise addition layer of the forward process can be the noised image corresponding to the sample image as described above.

[0088] In the image denoising process, the diffusion model can decompose the sample image into multiple image elements, and further, according to at least some of the multiple image elements, the noise image output by the last diffusion noise layer of the forward process is denoised to obtain a denoised image. Specifically, the image elements input into the diffusion model include global image elements and / or local image elements. The internal structure of the diffusion model can be as follows: Figure 7 or Fig. 9 The UNet structure shown in Figure 1 is as follows. Figure 7 or Fig. 9 The noisy image shown may be the noisy image output by the last diffusion noise layer of the forward process as described above. When the image elements input to the diffusion model include global image elements, the processing of the global image elements and the noisy image may refer to Figure 7 When the image elements input into the diffusion model include local image elements, the processing of the local image elements and the noise image can refer to Fig. 9 When the image elements input into the diffusion model include both global image elements and local image elements, the processing of the global image elements refers to Figure 7 As shown, the processing of local image elements refers to Fig. 9 As shown, at this time, Figure 7 and Fig. 9 can be merged together. In addition, Figure 7 and / or Fig. 9 The process shown can be an iterative process, and after multiple iterative denoising, a denoised image can be obtained.

[0089] Further, according to the similarity or difference between the denoised image and the sample image, the parameters of the diffusion model are adjusted so that the denoised image and the sample image gradually become similar in the subsequent iterative training process. Exemplarily, the denoising process can be guided according to the gradient data between the denoised image and the sample image. Among them, the original meaning of the gradient is a vector (vector), which means that the directional derivative of a function at this point achieves the maximum value along this direction, that is, the function at this point changes fastest along this direction (the direction of this gradient) and has the largest rate of change. It is also based on this principle that the direction of change of the denoised image can be guided to get closer to the sample image.

[0090] That is to say, after the sample image is decomposed into multiple image elements, a number of image elements are randomly selected from the multiple image elements to form a random subset. The diffusion model uses the subset as a condition to reconstruct the original image, i.e., the sample image, from the noisy image as much as possible.

[0091] This embodiment obtains a sample image and a noisy image corresponding to the sample image, and decomposes the sample image into multiple image elements. Further, the noisy image and at least some of the multiple image elements are input into the machine learning model to be trained, so that the machine learning model to be trained performs denoising on the noisy image according to at least some of the image elements to obtain a denoised image, and trains the machine learning model to be trained according to the sample image and the denoised image. The trained machine learning model can generate new images according to any combination of image elements, which greatly improves the diversity of images generated by the machine learning model, so that the diverse images can be widely used, for example, it can provide professional designers with rich and diverse images as design materials.

[0092] Specifically, the trained machine learning model as described above, such as the trained diffusion model, can be used to implement the image generation method described below.

[0093] Figure 4A flow chart of the image generation method provided in the embodiment of the present disclosure. The method can be executed by an image generation device, which can be implemented in software and / or hardware. The device can be configured in an electronic device, such as a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc. The server can specifically be a server in the cloud, and the image generation method can be executed by the cloud. Several computing nodes (cloud servers) can be deployed in the cloud, and each computing node has computing, storage and other processing resources. In the cloud, multiple computing nodes can be organized to provide a certain service. Of course, a computing node can also provide one or more services. The cloud can provide the service by providing a service interface to the outside, and the user calls the service interface to use the corresponding service. The service interface includes a software development kit (Software Development Kit, referred to as SDK), an application programming interface (Application Programming Interface, referred to as API), and the like. In addition, the image generation method described in this embodiment can be applicable to the following: Figure 2 The application scenario shown in Figure 1. Figure 2 As shown, the application scenario includes a terminal 21 and a server 22. After the server 22 trains a machine learning model such as a diffusion model through the model training method described above, the server 22 can retain the machine learning model locally so that the server 22 can generate a new image through the machine learning model. Alternatively, the server 22 can also deploy the machine learning model on the terminal 21 or other servers so that the terminal 21 or other servers can generate a new image through the machine learning model. The following is a detailed description of the example of the server 22 generating a new image through the machine learning model. Figure 4 As shown, the specific steps of this method are as follows:

[0094] S401: Acquire multiple image generation elements.

[0095] For example, a large number of sample images are stored in the local database of server 22 or the database corresponding to server 22. Assuming that a sample image can be decomposed into 8 different image elements, a large number of sample images can be decomposed into a large number of image elements. Further, server 22 can store the large number of image elements in another database. Alternatively, server 22 can store the large number of image elements in the database where the sample image is located, and establish a corresponding relationship between the sample image and the image element. For example, a corresponding relationship can be established between a sample image and the 8 different image elements decomposed therefrom. In addition, it can be understood that the large number of image elements can be divided into 8 categories of image elements, for example, feature vectors, depth maps, segmentation maps, sketches, grayscale maps, mask maps, color matching, and text descriptions are each considered a category. However, for two different sample images, the image elements of the same category are different. For example, the feature vector of sample image 1 is different from the feature vector of sample image 2, and the depth map of sample image 1 and the depth map of sample image 2 are two different depth maps.

[0096] Furthermore, the server 22 may randomly extract multiple image elements from the large number of image elements as multiple image generation elements. Specifically, the multiple image elements randomly extracted by the server 22 from the large number of image elements may come from the same sample image, or may come from multiple different sample images.

[0097] S402. Generate a target image according to the multiple image generating elements, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generating elements one-to-one, and the similarity between the corresponding target elements and the image generating elements is greater than or equal to a preset value.

[0098] Specifically, the server 22 can generate a target image according to the multiple image generation elements, that is, the multiple image generation elements are recombined (composed) to generate the target image. For example, the multiple image generation elements include the feature vector of sample image 1 and the depth map of sample image 2. In the process of generating the target image according to the feature vector of sample image 1 and the depth map of sample image 2, the server 22 can not only characterize the characteristic vector of the target image according to the feature vector of sample image 1, and characterize the depth map of the target image according to the depth map of sample image 2, but also automatically complete other image elements of the target image. For example, the server 22 can automatically complete the segmentation map, sketch, grayscale map, mask map, color matching and text description of the target image, so that the generated target image can also include 8 image elements. Assume that the feature vector of sample image 1 is recorded as A1, the depth map of sample image 2 is recorded as B1, and the feature vector, depth map, segmentation map, sketch, grayscale map, mask map, color scheme and text description of the target image are recorded as A, B, C, D, E, F, G respectively. Since the feature vector A of the target image and the feature vector A1 of sample image 1 belong to the same type of image elements, and the depth map B of the target image and the depth map B1 of sample image 2 belong to the same type of image elements, A and B can be recorded as target elements respectively. In addition, since the target elements are derived from the generation conditions, i.e., the multiple image generation elements, and the generated target image meets the requirements of the multiple image generation elements, the similarity between A and A1 is greater than or equal to the preset value, and the similarity between B and B1 is greater than or equal to the preset value. That is to say, when the feature vector A1 of sample image 1 and the depth map B1 of sample image 2 are used as conditions, the feature vector A of the generated target image generally meets the requirements of the feature vector A1, but statistically, there is a small difference between A and A1. However, in an ideal case, A and A1 can be the same. Similarly, there is a small difference between B and B1 or they are the same.

[0099] Optionally, generating a target image according to the multiple image generating elements includes: inputting the multiple image generating elements into a pre-trained machine learning model, so that the machine learning model generates the target image after iteratively denoising the noisy image according to the multiple image generating elements, and in the process of generating the target image, completing other image elements among the multiple image elements of the target image except the multiple target elements.

[0100] like Figure 5As shown, when the server 22 generates a target image according to the multiple image generation elements, the multiple image generation elements can be specifically input as multiple conditions into a pre-trained diffusion model, and the diffusion model mainly includes a denoising model, or the diffusion model is a denoising model. The diffusion model generates the target image after iteratively denoising the noisy image according to the multiple image generation elements. For example, the diffusion model performs 50 iterative denoising on the noisy image, and each iterative denoising is for a different noisy image. For example, in the first iterative denoising process, an image with Gaussian noise can be generated inside the diffusion model, and the image with Gaussian noise can be generated according to a known image, or can be generated according to a preset image size. For example, the known image is an image including 300*300 pixels, 300*300 is the size of the known image, and the pixel value on each pixel is between 0-255. The diffusion model can randomly extract a value from 0-255 as the new pixel value of each pixel in the 300*300 pixels, and the final 300*300 image is an image with Gaussian noise. Alternatively, without the existence of a known image, an image with Gaussian noise can also be generated according to a preset image size, such as 300*300. The generation process is similar to the aforementioned generation process and will not be repeated here. Further, the diffusion model can denoise the image with Gaussian noise according to the multiple conditions to obtain a denoised image, which is recorded as the first denoised image. In other words, the noisy image targeted by the first iterative denoising is an image with Gaussian noise generated internally by the diffusion model. In the second iterative denoising process, the diffusion model can add noise to the first denoised image to obtain a noisy image, and the diffusion model denoises the noisy image according to the multiple conditions to obtain a denoised image, which is recorded as the second denoised image. In other words, the noisy image targeted by the second iterative denoising is a noisy image after adding noise to the first denoised image. In some embodiments, the noise added to the first denoised image may be smaller than the Gaussian noise. Similarly, the noisy image targeted by the third iterative denoising is the noisy image after adding noise to the second denoised image. The noise added to the second denoised image may be smaller than the noise added to the first denoised image. That is to say, in each iterative denoising process, the added noise will gradually decay as the number of iterations increases. In addition, in other embodiments, it is not limited to adding noise according to the attenuation law. For example, in each iterative denoising process, the added noise may be random. The denoising is continuously iterated until 50 iterations have passed, and the denoised image output by the diffusion model is the target image.It can be understood that if the process of generating the target image according to the multiple image generation elements is regarded as a 50-iteration denoising process, that is, a round of iterative denoising process, then in the same round of iterative denoising process, the multiple image generation elements, that is, the multiple conditions can remain unchanged, so that the denoised image output by the diffusion model in each iteration of the same round can gradually meet the requirements of the multiple conditions as the number of iterations increases. For example, the multiple conditions are the feature vector of sample image 1 and the depth map of sample image 2, and the feature vector of the denoised image output by the diffusion model each time is constantly approaching the feature vector of sample image 1, and the depth map of the denoised image output by the diffusion model each time is constantly approaching the depth map of sample image 2, so that the target image not only meets the requirements of the feature vector of sample image 1, but also meets the requirements of the depth map of sample image 2. At the same time, in the process of iterative denoising, the diffusion model can also automatically complete other image elements of the target image, such as segmentation map, sketch, grayscale map, mask map, color matching and text description.

[0101] In addition, in some other embodiments, such as Figure 5 The denoising process shown can also be implemented within the diffusion model, for example, one iteration includes one denoising and one denoising. For denoising, the diffusion model can first predict the noise carried in the noisy image, and then subtract the noise from the noisy image. For denoising, the diffusion model can randomly collect a noise or determine a noise in a deterministic manner, and then add the noise to the denoised image. The noise determined in a deterministic manner can be, for example, the noise that gradually decays as the number of iterations increases as described above.

[0102] That is to say, after the diffusion model is trained, any image elements are combined into a condition set, so that the diffusion model uses the set as a condition, starting from pure random noise, that is, an image with Gaussian noise, and generates a target image that meets the conditions step by step.

[0103] The disclosed embodiment generates a target image through multiple image generation elements, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generation elements one by one, and the similarity between the corresponding target elements and the image generation elements is greater than or equal to a preset value. Since the number of image generation elements is large, and the combination of different image generation elements is also flexible and diverse, a large number of different target images can be generated by combining different multiple image generation elements, which greatly improves the diversity of target images.

[0104] In addition, since the machine learning model, such as the diffusion model, can complete other image elements in the multiple image elements of the target image except the multiple target elements in the process of generating the target image, therefore, when generating the target image through multiple image generation elements, it is not required that the types of the multiple image generation elements are all types of image elements of the target image, but can be some types of the total types, so that the input conditions of the diffusion model described in this embodiment allow the lack of some element types, thereby further improving the flexibility and diversity of target image generation. Assume that there are 100 sample images, each sample image is decomposed into 8 image elements, so that the 100 sample images correspond to 800 image elements, the 800 image elements include 8 types of image elements, one type of image elements has 100, and the combination of the 800 image elements has 100 to the 8th power. Each combination can generate a target image, so the number of target images, that is, the number of new images generated, can reach 100 to the 8th power, that is, a large number of new images can be combined through the image elements of the existing sample images. This phenomenon is called combinatorial explosion. Therefore, based on the combinatorial explosion, great freedom and customization capabilities are provided for generating customized images.

[0105] like Figure 6 As shown, after the sample image 30 is decomposed into 8 different image elements, multiple new images can be randomly or arbitrarily combined according to the image elements of the sample image 30 and the image elements of other sample images. For example, the depth map 32 and the segmentation map 33 of the sample image 30 are combined to form the shape of the sample image 30, 60 represents the feature vector of the first sample image, and a new image 61 can be generated by combining the shape of the sample image 30 and the feature vector 60 of the first sample image. The specific combination process can refer to the iterative denoising process inside the diffusion model as described above. Similarly, according to the mask map 36 of the sample image 30 and the text description 62 of the second sample image, a new image 63 is generated by combining. According to the feature vector 31 of the sample image 30 and the feature vector 64 of the third sample image, a new image 65 is generated by combining. According to the grayscale map 35 of the sample image 30 and the color matching 66 of the fourth sample image, a new image 67 is generated by combining.

[0106] Understandably, Figure 6 The above are just some schematic illustrations of the combination of the image elements of the sample image 30 and the image elements of other sample images. There are many other combinations, and thus, other new images can be derived. Figure 6The combination method of the image elements between the two sample images is only schematically illustrated. In other embodiments, the image elements of the sample image 30 are not limited to being combined with the image elements of another sample image. For example, the image elements of the sample image 30 can be combined with the image elements of two, three or more other sample images, which will not be described in detail here.

[0107] It can be seen that a large number of new images can be derived from any combination of image elements from at least two sample images. In some other embodiments, the derived new images can be further stored in the database as sample images.

[0108] In addition, the image elements to be combined may not only be directly derived from the sample image, for example, the image elements obtained by disassembling the sample image. In other embodiments, the image elements to be combined may also include image elements obtained by modifying or adjusting existing image elements, and / or image elements designed, drawn or defined by the user. Figure 6 As shown, after the sample image 30 is decomposed into 8 different image elements, the user can modify or adjust individual image elements or all of the 8 different image elements to obtain new image elements. In addition, the user can also design, draw or define image elements by himself, for example, the user can draw a sketch or define a text description. The following is an introduction in conjunction with a specific embodiment.

[0109] On the basis of the above embodiment, before acquiring multiple image generation elements, the method further includes: decomposing at least one sample image into multiple image elements respectively, and the multiple image elements respectively corresponding to the at least one sample image constitute a set.

[0110] For example, a large number of sample images are stored in the local database of the server 22 or in the database corresponding to the server 22. Assuming that a sample image can be decomposed into 8 different image elements, a large number of sample images can be decomposed into a large number of image elements. Further, the server 22 can form a set of the large number of image elements.

[0111] In a feasible implementation, obtaining multiple image generation elements includes: taking at least two image elements in the set as the multiple image generation elements; or taking a new image element and at least one image element in the set as the multiple image generation elements, the new image element being an image element input by a user or an image element after the user modifies the image element in the set; or taking at least two new image elements as the multiple image generation elements.

[0112] For example, when acquiring multiple image generating elements, the server 22 may randomly extract at least two image elements from the set as multiple image generating elements, wherein the at least two randomly extracted image elements may come from the same sample image or from different sample images.

[0113] Alternatively, the server 22 may obtain at least one new image element and randomly extract at least one image element from the set, so that the at least one new image element and the at least one randomly extracted image element constitute a plurality of image generation elements. Specifically, the at least one new image element includes an image element designed, drawn or defined by the user himself, and / or an image element obtained by modifying an image element in the set.

[0114] Alternatively, the server 22 may obtain at least two new image elements as the multiple image generation elements. The at least two new image elements include image elements designed, drawn or defined by the user himself, and / or image elements obtained by modifying image elements in the set.

[0115] In another feasible implementation, the method further includes: adding at least one new image element to the set to obtain an updated set, wherein the new image element is an image element input by a user or an image element after the user modifies the image element in the set; obtaining multiple image generation elements, including: using at least two image elements in the updated set as the multiple image generation elements.

[0116] For example, when the server 22 obtains at least one new image element, it adds the at least one new image element to the set as described above to obtain an updated set, and the updating process may be continuously updated. When the server 22 obtains multiple image generation elements, at least two image elements may be randomly extracted from the updated set as multiple image generation elements. At this time, the at least two randomly extracted image elements may both be image elements directly obtained after the sample image is disassembled, or may include both image elements directly obtained after the sample image is disassembled and new image elements, or may both be new image elements.

[0117] For example, each sample image is decomposed into 8 image elements, and 100 sample images correspond to 800 image elements. The 800 image elements include 8 types of image elements, and there are 100 image elements of one type. There are 100 to the power of 8 combinations of the 800 image elements.

[0118] If new image elements are added, the total number of image elements will likely be much greater than 800, and the number of combinations between image elements will be much greater than 100 to the power of 8. The number of new images generated by the combinations will be even larger, and the degree of combinatorial explosion will be higher.

[0119] Therefore, in this embodiment, the sample image is disassembled into multiple image elements, and then an artificial intelligence (AI) model such as a diffusion model is used to reassemble multiple different image elements into an image. In the process from disassembly to combination, the existing image elements can also be freely modified and edited, or the user can directly redesign, draw or define new image elements, thereby improving the controllability of the image element level. At the same time, when generating customized images, due to the higher flexibility of the combination between different image elements, the customized image has a higher degree of freedom and a stronger customization capability. Therefore, the paradigm of the combined generation model provided in this embodiment can realize extremely diverse controllable generation and editing capabilities based on a single framework, and has great potential in technical value and product application. In addition, the combined generation model is a single model that can support dozens of control modes. A small number of images can generate a large control space through the combination of control dimensions, which greatly improves the controllability.

[0120] Among the eight types of image elements mentioned above, some are global image elements in the image, such as color matching, image feature vectors, etc., while some are local image elements in the image, such as depth map, segmentation map, mask map, etc. Therefore, image elements can be divided into global image elements and local image elements. The following introduces the process of how the diffusion model processes global image elements and local image elements respectively, combining the internal structural characteristics of the diffusion model.

[0121] Based on the above embodiment, the machine learning model includes a first intermediate layer, a second intermediate layer, a third intermediate layer and a fourth intermediate layer, the second intermediate layer is the next layer of the first intermediate layer, the fourth intermediate layer is the next layer of the third intermediate layer, and the fourth intermediate layer is located above the first intermediate layer.

[0122] Specifically, the diffusion model can be implemented through a trainable deep neural network, whose structure is a deep understanding of deep learning segmentation network (Convolutional Networks for Biomedical Image Segmentation, UNet). Figure 7The following is a schematic diagram of the Unet structure. Specifically, UNet includes an input layer, a normalization layer, a scale shift layer, a multi-head cross-attention layer, and other layers not described in the previous section. Figure 7 The scale change layer is recorded as the first intermediate layer, the normalization layer is recorded as the second intermediate layer, the multi-head cross attention layer is recorded as the third intermediate layer, and the intermediate layer adjacent to the multi-head cross attention layer and located below the multi-head cross attention layer is recorded as the fourth intermediate layer. Figure 7 As shown from bottom to top, the second intermediate layer is the next layer of the first intermediate layer, the fourth intermediate layer is the next layer of the third intermediate layer, and the fourth intermediate layer is located above the first intermediate layer. For example, the fourth intermediate layer may be the upper layer or several layers above the first intermediate layer.

[0123] Optionally, the multiple image generation elements include at least one global image generation element, and the global image generation element corresponds to a global vector.

[0124] Specifically, the input of the diffusion model is a plurality of conditions, such as the plurality of image generation elements as described above, and the output of the diffusion model is a target image, i.e., a generated new image. The plurality of image generation elements may be a plurality of image elements. At least one global image generation element included in the plurality of image generation elements is recorded as a global image element. A global image element may be represented by a vector, which is recorded as a global vector.

[0125] Optionally, the machine learning model iteratively denoises the noisy image according to the multiple image generation elements, including: Figure 8 The following steps are shown:

[0126] S801. Add at least one global vector and a vector corresponding to this iteration to obtain a first addition result.

[0127] Assuming that multiple image generation elements include multiple global image elements, the dimensions of the global vectors corresponding to different global image elements may be different. Therefore, each global vector is mapped to another vector through several layers of fully connected layers, so that the dimensions of each mapped vector are the same, and each mapped vector is located in the same coordinate space. Then each mapped vector is added, and the result of the addition is added to the vector corresponding to the accumulated number of iterations to obtain the first addition result. Among them, the accumulated number of iterations indicates the number of iterative denoising that has been performed so far, and the accumulated number of iterations can also indicate the number of iterative denoising that is being performed this time. For example, each time an iterative denoising is completed, the accumulated number of iterations is increased by 1. Therefore, each accumulated number of iterations is a different value, and different values ​​can be represented by different vectors.

[0128] If the multiple image generation elements include a global image element, the global vector corresponding to the global image element is mapped to another vector through several layers of fully connected layers, and then the mapped vector and the vector corresponding to the accumulated number of iterations are added to obtain a first addition result.

[0129] S802. Input the first addition result into the first intermediate layer, where the first intermediate layer is used to map the first addition result into a first vector and a second vector, where output information of the first intermediate layer is determined based on the first vector, output information of the second intermediate layer and the second vector, and where output information of the second intermediate layer is related to the noisy image.

[0130] like Figure 7 As shown, the first addition result can be input into the scale change layer, thereby realizing the introduction of the global vector into UNet. The scale change layer can map the first addition result into a first vector and a second vector. Specifically, the first vector is recorded as a Scale vector, and the second vector is recorded as a Shift vector.

[0131] like Figure 7As shown, assuming that the height of the noisy image is denoted as H and the width is denoted as W, the noisy image includes H*W pixels, each pixel can be represented by a vector, and the dimension of the vector can be N dimensions. Therefore, the noisy image can be represented by H*W vectors. Further, the H*W vectors are input into UNet, so that each intermediate layer in UNet can process the H*W vectors. During the processing, the information output by each intermediate layer is H*W vectors, but the vectors output by different intermediate layers may change, and the dimension of the vector may also change. Assume that the dimension of each of the H*W vectors output by the normalization layer is N1. When the normalization layer inputs the H*W vectors into the scale change layer, the scale change layer can process each of the H*W vectors according to the Scale vector and Shift vector as described above. For example, each of the H*W vectors can be recorded as a target vector, and the scale change layer can perform the calculation shown in the following formula (1):

[0132] Scale vector * target vector + shift vector (1)

[0133] Therefore, the output information of the scale change layer is also H*W vectors, and each vector of the output of the scale change layer is calculated according to formula (1).

[0134] S803: Map the at least one global vector into multiple vectors to obtain a vector sequence.

[0135] In addition, Figure 7 In the above, the global vector of the global image element can also be implied into multiple vectors. For example, the multiple image generation elements include 5 global image elements, and the 5 global image elements correspond to 5 global vectors. After each global vector is implied into 4 vectors, 20 vectors can be obtained. The 20 vectors constitute a vector sequence.

[0136] S804: Input the vector sequence into the third intermediate layer, where the third intermediate layer is used to interact the vector sequence with output information of the fourth intermediate layer, so that the output information of the third intermediate layer includes information of the vector sequence.

[0137] like Figure 7 As shown, the vector sequence can be input into the multi-head cross attention layer. In addition, the output information of the next layer of the multi-head cross attention layer is H*W vectors. When the H*W vectors are input into the multi-head cross attention layer, the multi-head cross attention layer can control the vector sequence and the H*W vectors to interact, so that each of the H*W vectors output by the multi-head cross attention layer contains the information of the vector sequence.

[0138] S805. Complete this iterative denoising according to the output information of the third intermediate layer.

[0139] For example Figure 7 As shown, the H*W vectors output by the multi-head cross-attention layer can continue to be processed by the intermediate layer above the multi-head cross-attention layer until the top intermediate layer inside UNet outputs H*W vectors, and each vector in the H*W vectors is replaced by a corresponding pixel point to obtain a H*W image, which is the denoised image output by UNet after completing this iterative denoising.

[0140] In some other embodiments, the multiple image generating elements include at least one local image generating element, and the local image generating element corresponds to a local vector; the multiple image generating elements are input into a pre-trained machine learning model, including: adding at least one local vector to obtain a second addition result; splicing the second addition result and the representation vector of the noisy image to obtain a splicing result; and inputting the splicing result into the pre-trained machine learning model.

[0141] For example, at least one local image generation element included in the plurality of image generation elements is recorded as a local image element. A local image element can be represented by a vector, which is recorded as a local vector.

[0142] When multiple image generation elements include multiple local image elements, if the sizes of the multiple local image elements are different, for example, the initial sizes of the segmentation map and the sketch are different, the sizes of the multiple local image elements are first adjusted so that the sizes of the multiple local image elements are the same. In addition, the dimensions of local vectors corresponding to different local image elements may be different. For example, the size of the segmentation map and the size of the sketch are both H*W, and the segmentation map and the sketch can be represented by H*W vectors respectively, and one vector represents one pixel. However, the dimension of each vector corresponding to the segmentation map and the dimension of each vector corresponding to the sketch may be different. At this time, through Fig. 9The convolution layer (Convolution) shown maps the local vector corresponding to each local image element to a vector of equal dimension, so that the dimension of each vector corresponding to the segmentation map is the same as the dimension of each vector corresponding to the sketch. Further, each local vector after mapping is added, for example, the H*W vector representations corresponding to the segmentation map and the H*W vector representations corresponding to the sketch are added. Specifically, the vector representations corresponding to the same pixel are added to obtain a second addition result, and the second addition result is an H*W vector representation, but the representation vector before the addition is different from the representation vector after the addition. Further, the second addition result and the representation vector of the noisy image (for example, H*W vector representations) are spliced ​​to obtain a splicing result; the splicing result is input into UNet. Thereby, the process of introducing local vectors into UNet is realized.

[0143] When multiple image generation elements include a local image element, the local vector corresponding to the local image element is processed by the convolution layer, and the processing result is directly spliced ​​with the representation vector of the noisy image to obtain a splicing result; further, the splicing result is input into UNet, thereby realizing the process of introducing the local vector into UNet.

[0144] It is understandable that if multiple image generation elements are global image generation elements, the processing inside UNet is as follows: Figure 7 If multiple image generation elements are local image generation elements, the processing inside UNet is as follows: Fig. 9 If multiple image generation elements include both global image generation elements and local image generation elements, the internal processing of UNet is Figure 7 and Fig. 9 The superposition of Figure 7 As shown, the global image generation elements are processed, refer to Fig. 9 As shown, local image generation elements are processed.

[0145] In addition, once UNet is trained, it can be directly used for more than a dozen image generation or editing operations such as style transfer, image-to-image translation, image editing based on sketches, segmentation maps, depth maps, etc., and color matching without fine-tuning. Among them, style transfer refers to taking the content of one image and the style of another image as conditions, and then iteratively denoising the image with Gaussian noise based on the conditions. Image-to-image translation can be taking the feature vector of one image and the text description of another image as conditions, and then iteratively denoising the image with Gaussian noise based on the conditions. Fig.10As shown in the figure, the small images under each large image are the conditions of the large image. For example, the three small images under large image 61 are the conditions used by UNet when generating large image 61; the two small images under large image 63 are the conditions used by UNet when generating large image 63; the two small images under large image 65 are the conditions used by UNet when generating large image 65; and the two small images under large image 67 are the conditions used by UNet when generating large image 67.

[0146] It is understandable that the essence of UNet is to predict noise. To denoise a noisy image, UNet actually predicts the noise carried in the noisy image and then subtracts the noise from the noisy image. If UNet adds a random noise or a noise determined in a deterministic way to the original image, it is equivalent to adding noise to the original image. Among them, the noise determined in a deterministic way can be, for example, the noise that gradually decays with the increase of the number of iterations as described above. Therefore, UNet can not only achieve denoising, but also add noise.

[0147] Image editing based on sketches, segmentation maps, depth maps, etc. can be to first convert an image into a noisy image, and then convert the noisy image into other images. The two conversion processes can be implemented by UNet respectively, but the conditions used in the two conversion processes are different. For example, the condition used to convert an image into a noisy image is the image element of the image, that is, the condition is related to the image. The condition used to convert the noisy image into other images is not related to the image. Fig.11 As shown, image 100 represents a noisy image converted by UNet from an original image, and image 101 represents another image converted by UNet from the noisy image. The conditions used when generating image 100 are the image elements of the original image. The conditions used when generating image 101 are not related to the original image, for example, the conditions used are other image elements except the image elements of the original image.

[0148] Fig.12 FIG. 1 is a flow chart of an image generation method provided by another embodiment of the present disclosure. In this embodiment, the method can be executed by a cloud, which includes: Fig.13 The cloud server 122 is shown. The specific steps of the method are as follows:

[0149] S1101. Receive multiple image generation elements sent by a terminal.

[0150] For example Fig.13As shown, the cloud server 122 receives multiple image generation elements from the terminal 121. The multiple image generation elements may all be image elements directly obtained after the sample image is disassembled, or may include both image elements directly obtained after the sample image is disassembled and new image elements, or may all be new image elements. The new image element is an image element input by the user or an image element modified by the user on an existing image element.

[0151] S1102. Generate a target image according to the multiple image generating elements, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generating elements one-to-one, and the similarity between the corresponding target elements and the image generating elements is greater than or equal to a preset value.

[0152] Specifically, the cloud server 122 can generate a target image based on the multiple image generation elements, that is, the multiple image generation elements are recombined to generate the target image. Specifically, the cloud server 122 can input the multiple image generation elements as multiple conditions into a pre-trained diffusion model, and the diffusion model generates the target image after iteratively denoising the noisy image based on the multiple image generation elements.

[0153] S1103: Feedback the target image to the terminal.

[0154] For example, the cloud server 122 may feed back the target image to the terminal 121 .

[0155] The disclosed embodiment generates a target image through multiple image generation elements, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generation elements one by one, and the similarity between the corresponding target elements and the image generation elements is greater than or equal to a preset value. Since the number of image generation elements is large, and the combination of different image generation elements is also flexible and diverse, a large number of different target images can be generated by combining different multiple image generation elements, which greatly improves the diversity of target images.

[0156] Fig.14 This is a flow chart of an image generation method provided by another embodiment of the present disclosure. In this embodiment, the method can be executed by a terminal. Specifically, the method has the following steps:

[0157] S1301. In response to multiple image generating elements selected by a user, a target image is generated, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generating elements one-to-one, and the similarity between the corresponding target elements and the image generating elements is greater than or equal to a preset value.

[0158] For example Fig.15 As shown, six image elements are displayed on the user interface of terminal 121, and the user can select several image elements from the six image elements as image generation elements. For example, the user can select three image elements in the first row of the user interface as image generation elements.

[0159] If the terminal 121 is deployed with a diffusion model that has been trained as described above, the terminal 121 can respond to multiple image generation elements selected by the user and input the multiple image generation elements into the diffusion model, so that the diffusion model iteratively denoises the noisy image according to the multiple image generation elements and generates a target image 61.

[0160] S1302: Display the target image.

[0161] Furthermore, the terminal 121 may display the target image 61 on a user interface.

[0162] It is understandable that if the terminal 121 is not deployed with the diffusion model trained as described above, the terminal 121 may send the multiple image generation elements selected by the user to the terminal 121. Fig.13 The cloud server 122 shown generates a target image based on the multiple image generation elements, and then feeds the target image back to the terminal 121, which then displays the target image on the user interface.

[0163] Fig.16 FIG. 1 is a flow chart of an image generation method provided by another embodiment of the present disclosure. In this embodiment, the method can be executed by a cloud, which includes: Fig.13 The cloud server 122 is shown. The specific steps of the method are as follows:

[0164] S1501: Decompose at least one sample image into a plurality of image elements.

[0165] like Fig.17 As shown, the cloud server 122 decomposes at least one sample image into a plurality of image elements.

[0166] S1502: Construct a database according to a plurality of image elements respectively corresponding to the at least one sample image and identifiers of the image elements.

[0167] For example Fig.17 In step 1 shown, the cloud server 122 may send the decomposed image elements and the identifiers of the image elements to the database server, so that the database server constructs a database locally.

[0168] S1503. Receive multiple identifiers sent by the terminal.

[0169] like Fig.17 In steps 2, 3, and 4 shown, the terminal 121 may send a connection request to the database server, for example, a Hyper Text Transfer Protocol (HTTP) request. Further, the database server feeds back web page data to the terminal 121, and the terminal 121 displays a web page according to the web page data, and the web page includes image elements stored in the database. Further, the user of the terminal 121 may modify or adjust the existing image elements on the web page. Or the user may directly add image elements designed and drawn by himself to the web page. Thus, the database is updated. In addition, the user may also select image elements in the web page, for example, the user selects 3 image elements in the first row of the web page. At this time, the terminal 121 may send the identifiers corresponding to the 3 image elements to the cloud server 122, so that the cloud server 122 receives multiple identifiers.

[0170] S1504. Acquire, according to the multiple identifiers, multiple image elements corresponding to the multiple identifiers from the database.

[0171] like Fig.17 In steps 5 and 6 shown, the cloud server 122 may send the multiple identifiers to the database server, and the database server may obtain multiple image elements corresponding to the multiple identifiers from the local database according to the multiple identifiers, for example, one identifier corresponds to one image element. Further, the database server sends the multiple image elements to the cloud server 122.

[0172] S1505. Generate a target image based on multiple image elements corresponding to the multiple identifiers, so that the multiple image elements of the target image include multiple target elements, the multiple image elements corresponding to the multiple identifiers correspond to the multiple target elements one-to-one, and the similarity between the corresponding target elements and the image elements is greater than or equal to a preset value.

[0173] For example Fig.17 In steps 7 and 8 shown, the cloud server 122 can generate a target image based on the multiple image elements. The generation process can refer to the process described above and will not be repeated here. Further, the cloud server 122 can feed back the target image to the terminal 121. The terminal 121 displays the target image, such as the target image 61, on its user interface.

[0174] Fig.18 The image generating device provided by the embodiment of the present disclosure can execute the processing flow provided by the image generating method embodiment, such as Fig.18As shown, the image generating device 170 includes:

[0175] An acquisition module 171, used to acquire multiple image generation elements;

[0176] A generating module 172 is used to generate a target image according to the multiple image generating elements, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generating elements one by one, and the similarity between the corresponding target elements and the image generating elements is greater than or equal to a preset value.

[0177] Optionally, when generating a target image based on the multiple image generating elements, the generating module 172 is specifically used to: input the multiple image generating elements into a pre-trained machine learning model, so that the machine learning model iteratively denoises the noisy image based on the multiple image generating elements to generate the target image, and in the process of generating the target image, complete the multiple image elements of the target image except the multiple target elements.

[0178] Optionally, the machine learning model includes a first intermediate layer, a second intermediate layer, a third intermediate layer and a fourth intermediate layer, the second intermediate layer is a layer below the first intermediate layer, the fourth intermediate layer is a layer below the third intermediate layer, and the fourth intermediate layer is located above the first intermediate layer;

[0179] The plurality of image generation elements include at least one global image generation element, and the global image generation element corresponds to a global vector;

[0180] When the machine learning model iteratively denoises the noisy image according to the multiple image generation elements, it is specifically used to:

[0181] Adding at least one global vector and a vector corresponding to this iteration to obtain a first addition result;

[0182] inputting the first addition result into the first intermediate layer, the first intermediate layer being used to map the first addition result into a first vector and a second vector, output information of the first intermediate layer being determined according to the first vector, output information of the second intermediate layer and the second vector, and output information of the second intermediate layer being related to the noisy image;

[0183] Mapping the at least one global vector into a plurality of vectors respectively to obtain a vector sequence;

[0184] Inputting the vector sequence into the third intermediate layer, wherein the third intermediate layer is used to interact the vector sequence with the output information of the fourth intermediate layer, so that the output information of the third intermediate layer includes information of the vector sequence;

[0185] This iterative denoising is completed according to the output information of the third intermediate layer.

[0186] Optionally, the multiple image generation elements include at least one local image generation element, and the local image generation element corresponds to a local vector; when the generation module 172 inputs the multiple image generation elements into the pre-trained machine learning model, it is specifically used to:

[0187] Adding at least one local vector to obtain a second addition result;

[0188] splicing the second addition result and the representation vector of the noisy image to obtain a splicing result;

[0189] The splicing result is input into the pre-trained machine learning model.

[0190] Optionally, the device 170 further includes a disassembling module 173, which is used to disassemble at least one sample image into a plurality of image elements before the acquisition module 171 acquires a plurality of image generation elements, and the plurality of image elements corresponding to the at least one sample image constitute a set.

[0191] Optionally, when acquiring multiple image generation elements, the acquisition module 171 is specifically used to:

[0192] Using at least two image elements in the set as the multiple image generation elements; or

[0193] Using a new image element and at least one image element in the set as the multiple image generation elements, the new image element being an image element input by a user or an image element obtained by modifying the image element in the set by the user; or

[0194] At least two new image elements are used as the plurality of image generation elements.

[0195] Optionally, the device 170 also includes an updating module 174, which is used to add at least one new image element to the set to obtain an updated set, wherein the new image element is an image element input by a user or an image element after the user modifies the image element in the set; in this case, when acquiring multiple image generation elements, the acquisition module 171 is specifically used to: use at least two image elements in the updated set as the multiple image generation elements.

[0196] Fig.18 The image generating device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment, and its implementation principle and technical effect are similar and will not be described in detail here.

[0197] Fig.19 The schematic diagram of the structure of the model training device provided in the embodiment of the present disclosure. The model training device provided in the embodiment of the present disclosure can execute the processing flow provided in the embodiment of the model training method, such as Fig.19 As shown, the model training device 180 includes:

[0198] An acquisition module 181 is used to acquire a sample image and a noisy image corresponding to the sample image;

[0199] A disassembly module 182, used for disassembling the sample image into multiple image elements;

[0200] An input module 183, configured to input the noisy image and at least some of the multiple image elements into a machine learning model to be trained, so that the machine learning model to be trained performs denoising processing on the noisy image according to the at least some of the image elements to obtain a denoised image;

[0201] The training module 184 is used to train the machine learning model to be trained according to the sample image and the denoised image to obtain a trained machine learning model, and the trained machine learning model is used to implement the image generation method as described above.

[0202] Fig.19 The model training device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0203] The above describes the internal functions and structure of the image generation and model training device, which can be implemented as an electronic device. Fig. 20 This is a schematic diagram of the structure of an electronic device embodiment provided by the present disclosure. Fig. 20 As shown, the electronic device includes a memory 191 and a processor 192 .

[0204] The memory 191 is used to store programs. In addition to the above-mentioned programs, the memory 191 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0205] Memory 191 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0206] The processor 192 is coupled to the memory 191 and executes the program stored in the memory 191 to:

[0207] Get multiple image generation elements;

[0208] A target image is generated based on the multiple image generating elements, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generating elements one-to-one, and the similarity between the corresponding target elements and the image generating elements is greater than or equal to a preset value.

[0209] Alternatively, the processor 192 is further configured to:

[0210] receiving a plurality of image generation elements sent by a terminal;

[0211] Generate a target image according to the multiple image generating elements, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generating elements one by one, and the similarity between the corresponding target elements and the image generating elements is greater than or equal to a preset value;

[0212] The target image is fed back to the terminal.

[0213] Alternatively, the processor 192 is further configured to:

[0214] In response to the plurality of image generating elements selected by the user, a target image is generated, so that the plurality of image elements of the target image include a plurality of target elements, the plurality of target elements correspond to the plurality of image generating elements one by one, and the similarity between the corresponding target elements and the image generating elements is greater than or equal to a preset value;

[0215] The target image is displayed.

[0216] Alternatively, the processor 192 is further configured to:

[0217] Decomposing at least one sample image into a plurality of image elements respectively;

[0218] constructing a database according to a plurality of image elements respectively corresponding to the at least one sample image and identifiers of the image elements;

[0219] receiving multiple identifiers sent by a terminal;

[0220] According to the multiple identifiers, acquiring multiple image elements corresponding to the multiple identifiers from the database;

[0221] A target image is generated based on multiple image elements corresponding to the multiple identifiers, so that the multiple image elements of the target image include multiple target elements, the multiple image elements corresponding to the multiple identifiers correspond to the multiple target elements one-to-one, and the similarity between the corresponding target elements and the image elements is greater than or equal to a preset value.

[0222] Alternatively, the processor 192 is further configured to:

[0223] Acquire a sample image and a noisy image corresponding to the sample image;

[0224] Decompose the sample image into multiple image elements;

[0225] Inputting the noisy image and at least part of the multiple image elements into a machine learning model to be trained, so that the machine learning model to be trained performs denoising processing on the noisy image according to the at least part of the image elements to obtain a denoised image;

[0226] According to the sample image and the denoised image, the machine learning model to be trained is trained to obtain a trained machine learning model, and the trained machine learning model is used to implement the image generation method as described above.

[0227] Further, if Fig. 20 As shown, the electronic device may also include: a communication component 193, a power component 194, an audio component 195, a display 196 and other components. Fig. 20 Only some components are shown schematically, which does not mean that the electronic device only includes Fig. 20 Components shown.

[0228] The communication component 193 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 193 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 193 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0229] The power supply component 194 provides power to various components of the electronic device. The power supply component 194 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0230] The audio component 195 is configured to output and / or input audio signals. For example, the audio component 195 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 191 or sent via the communication component 193. In some embodiments, the audio component 195 also includes a speaker for outputting audio signals.

[0231] The display 196 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0232] In addition, an embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the image generation method or model training method described in the above embodiment.

[0233] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0234] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating an image, wherein: The method comprises: Acquire a plurality of image generation elements, wherein any one of the image generation elements is determined by different types of image elements, the different types of image elements are obtained by decomposing a sample image, and the different types of image elements are used to describe different aspects of information of the sample image; Generate a target image according to the multiple image generating elements, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generating elements one by one, and the similarity between the corresponding target elements and the image generating elements is greater than or equal to a preset value; Among them, generating a target image according to the multiple image generating elements includes: inputting the multiple image generating elements into a pre-trained machine learning model, so that the machine learning model generates the target image after iteratively denoising the noisy image according to the multiple image generating elements.

2. The method according to claim 1, wherein: Generating a target image according to the plurality of image generation elements further comprises: In the process of generating the target image, other image elements among the multiple image elements of the target image except the multiple target elements are completed.

3. The method according to claim 2, wherein: The machine learning model includes a first intermediate layer, a second intermediate layer, a third intermediate layer and a fourth intermediate layer, wherein the second intermediate layer is a layer below the first intermediate layer, the fourth intermediate layer is a layer below the third intermediate layer, and the fourth intermediate layer is located above the first intermediate layer; The plurality of image generation elements include at least one global image generation element, and the global image generation element corresponds to a global vector; The machine learning model iteratively denoises the noisy image according to the multiple image generation elements, including: Adding at least one global vector and a vector corresponding to this iteration to obtain a first addition result; inputting the first addition result into the first intermediate layer, the first intermediate layer being used to map the first addition result into a first vector and a second vector, output information of the first intermediate layer being determined according to the first vector, output information of the second intermediate layer and the second vector, and output information of the second intermediate layer being related to the noisy image; Mapping the at least one global vector into a plurality of vectors respectively to obtain a vector sequence; Inputting the vector sequence into the third intermediate layer, wherein the third intermediate layer is used to interact the vector sequence with the output information of the fourth intermediate layer, so that the output information of the third intermediate layer includes information of the vector sequence; This iterative denoising is completed according to the output information of the third intermediate layer.

4. The method according to claim 2, wherein: The plurality of image generation elements include at least one local image generation element, and the local image generation element corresponds to a local vector; Inputting the multiple image generation elements into a pre-trained machine learning model includes: Adding at least one local vector to obtain a second addition result; splicing the second addition result and the representation vector of the noisy image to obtain a splicing result; The splicing result is input into the pre-trained machine learning model.

5. The method according to claim 1, wherein: Before acquiring a plurality of image generation elements, the method further includes: At least one sample image is decomposed into a plurality of image elements, and the plurality of image elements corresponding to the at least one sample image constitute a set.

6. The method according to claim 5, wherein: Get multiple image generation elements, including: Using at least two image elements in the set as the multiple image generation elements; or Using a new image element and at least one image element in the set as the multiple image generation elements, the new image element being an image element input by a user or an image element obtained by modifying the image element in the set by the user; or At least two new image elements are used as the plurality of image generation elements.

7. The method according to claim 5, wherein: The method further comprises: Adding at least one new image element to the set to obtain an updated set, wherein the new image element is an image element input by a user or an image element after the user modifies the image element in the set; Get multiple image generation elements, including: At least two image elements in the updated set are used as the multiple image generation elements.

8. A method for generating an image, wherein: The method comprises: A plurality of image generation elements sent by a receiving terminal, wherein any one of the image generation elements is determined by different types of image elements, the different types of image elements are obtained by decomposing a sample image, and the different types of image elements are used to describe different aspects of information of the sample image; Generate a target image according to the multiple image generating elements, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generating elements one by one, and the similarity between the corresponding target elements and the image generating elements is greater than or equal to a preset value; The step of generating a target image according to the plurality of image generating elements comprises: inputting the plurality of image generating elements into a pre-trained machine learning model, so that the machine learning model generates the target image after iteratively denoising the noisy image according to the plurality of image generating elements; The target image is fed back to the terminal.

9. A method for generating an image, wherein: The method comprises: In response to multiple image generating elements selected by a user, a target image is generated, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generating elements one by one, and the similarity between the target elements and the image generating elements that have a corresponding relationship is greater than or equal to a preset value, wherein any image generating element is determined by different types of image elements, the different types of image elements are obtained by disassembling a sample image, and the different types of image elements are used to describe different aspects of information of the sample image, and in response to the multiple image generating elements selected by the user, the target image is generated, including: inputting the multiple image generating elements into a pre-trained machine learning model, so that the machine learning model generates the target image after iteratively denoising the noisy image according to the multiple image generating elements; The target image is displayed.

10. A method for generating an image, wherein: The method comprises: Decomposing at least one sample image into a plurality of image elements, wherein different image elements are used to describe different aspects of information of the at least one sample image; constructing a database according to a plurality of image elements respectively corresponding to the at least one sample image and identifiers of the image elements; receiving multiple identifiers sent by a terminal; According to the multiple identifiers, acquiring multiple image elements corresponding to the multiple identifiers from the database; Generate a target image according to the multiple image elements corresponding to the multiple identifiers, so that the multiple image elements of the target image include multiple target elements, the multiple image elements corresponding to the multiple identifiers correspond to the multiple target elements one-to-one, and the similarity between the corresponding target elements and the image elements is greater than or equal to a preset value; Among them, generating a target image according to multiple image elements corresponding to the multiple identifiers includes: inputting the multiple image elements into a pre-trained machine learning model, so that the machine learning model iteratively denoises the noisy image according to the multiple image elements to generate the target image.

11. A model training method, wherein: The method comprises: Acquire a sample image and a noisy image corresponding to the sample image; Decomposing the sample image into a plurality of image elements, wherein different image elements are used to describe different aspects of information of the sample image; Inputting the noisy image and at least part of the multiple image elements into a machine learning model to be trained, so that the machine learning model to be trained performs denoising processing on the noisy image according to the at least part of the image elements to obtain a denoised image; The machine learning model to be trained is trained according to the sample image and the denoised image to obtain a trained machine learning model, and the trained machine learning model is used to implement the method as described in any one of claims 1 to 10.

12. An image generating device, wherein: include: An acquisition module, used for acquiring a plurality of image generation elements, wherein any image generation element is determined by different types of image elements, the different types of image elements are obtained by disassembling a sample image, and the different types of image elements are used for describing different aspects of information of the sample image; A generation module is used to generate a target image according to the multiple image generation elements, so that the multiple image elements of the target image include multiple target elements, the multiple target elements correspond to the multiple image generation elements one-to-one, and the similarity between the target elements and the image generation elements in a corresponding relationship is greater than or equal to a preset value, wherein the target image is generated according to the multiple image generation elements, including: inputting the multiple image generation elements into a pre-trained machine learning model, so that the machine learning model generates the target image after iteratively denoising the noisy image according to the multiple image generation elements.

13. An electronic device, wherein: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images