Image processing method and apparatus, and computer device, storage medium and program product

Abstract

Provided in the present application are an image processing method and apparatus, and a computer device, a storage medium and a program product. The method relates to computer vision technology of artificial intelligence. The method comprises: acquiring sample data, wherein the sample data comprises a sample image, a sample category to which the sample image belongs, and a sample identifier, which is set for the sample image; calling an image generation model to extract training representation information of the sample identifier; calling the image generation model to predict, on the basis of the training representation information of the sample identifier, noise added to a noise image corresponding to the sample image, so as to obtain predicted noise, wherein the noise image is obtained by adding label noise to the sample image; and on the basis of the difference between the label noise and the predicted noise, and the difference between training semantics of the sample identifier and semantics of the sample category, updating model parameters of the image generation model, so as to train the image generation model. Using the present application can reduce the degree of overfitting of the image generation model and improve the training effect of the image generation model.

Description

Image processing method, device, computer equipment, storage medium, and program product

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 28, 2023, with application number 202311098890.X and application name “Image processing method, device and computer equipment, storage medium, program product”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, in particular to the field of artificial intelligence technology, and specifically to an image processing method, an image processing apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0003] With the rapid development of artificial intelligence technology, image generation models (also known as diffusion models) have made significant progress in the field of image generation, unleashing the creative potential of artificial intelligence. The application scenarios of image generation models are generally as follows: specific features (for example, a specific dog in an image) can be embedded into the image generation model to generate diffusion images related to the specific features (for example, a specific dog in an image standing on the beach). Ideally, multiple sample images of the same category are required to train the image generation model. However, in reality, multiple sample images of the same category are often lacking, and a single sample image is generally used to train the image generation model. However, using a single sample image to train the image generation model often leads to the problem of overfitting, resulting in poor training results for the image generation model.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide an image processing method, apparatus, computer equipment, storage medium, and program product, which can reduce the overfitting degree of the image generation model and improve the training effect of the image generation model.

[0006] In one aspect, an embodiment of the present application provides an image processing method, the image processing method comprising:

[0007] Obtaining sample data for training an image generation model, the sample data including a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image;

[0008] Calling the image generation model to extract training representation information of the sample identifier, the training representation information of the sample identifier is used to represent the training semantics of the sample identifier, and the training semantics of the sample identifier is used to express the sample characteristics of the sample image under the sample category;

[0009] The image generation model is called based on the training representation information of the sample identification to predict the noise added to the noise image corresponding to the sample image to obtain the predicted noise; the noise image is obtained by adding noise to the sample image, and the noise added by the noise addition is the label noise;

[0010] Based on the difference between the labeled noise and the predicted noise, as well as the difference between the training semantics of the sample identifier and the semantics of the sample category, the model parameters of the image generation model are updated to train the image generation model; the trained image generation model is used to denoise the noisy image based on the text description information containing the sample identifier, and generate a diffusion image that is semantically related to the text description information.

[0011] Accordingly, an embodiment of the present application provides an image processing device, comprising:

[0012] an acquisition unit, configured to acquire sample data for training an image generation model, the sample data including a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image;

[0013] a processing unit, configured to call an image generation model to extract training representation information of a sample identifier, wherein the training representation information of the sample identifier is used to represent the training semantics of the sample identifier, and the training semantics of the sample identifier is used to express sample features of the sample image under the sample category;

[0014] The processing unit is further configured to call the image generation model based on the training representation information of the sample identification, predict the noise added to the noise image corresponding to the sample image, and obtain the predicted noise; the noise image is obtained by performing noise processing on the sample image, and the noise added by the noise processing is the label noise;

[0015] The processing unit is further used to update the model parameters of the image generation model based on the difference between the labeled noise and the predicted noise, and the difference between the training semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model; the trained image generation model is used to denoise the noisy image based on the text description information containing the sample identifier, and generate a diffusion image that is semantically related to the text description information.

[0016] Accordingly, an embodiment of the present application provides a computer device, comprising:

[0017] a processor suitable for implementing a computer program;

[0018] A computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the above-mentioned image processing method.

[0019] Accordingly, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is read and executed by a processor of a computer device, the computer device executes the above-mentioned image processing method.

[0020] Accordingly, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the above-mentioned image processing method.

[0021] In an embodiment of the present application, by extracting the training representation information of the sample identifier, the semantics of the sample identifier during the training process can be obtained; the training representation information of the sample identifier can be used to denoise the noisy image corresponding to the sample image to obtain predicted noise; the noisy image corresponding to the sample image is obtained by adding labeled noise to the sample image, and based on the difference between the labeled noise and the predicted noise, the model parameters of the image generation model are updated, and the image generation model is trained, which can improve the image generation ability of the image generation model and improve the training effect of the image generation model. In the process of training the image generation model using a single sample image, a sample identifier can be set for the sample image, and the training semantics of the sample identifier can be used to express the sample characteristics of the sample image under the sample category. Based on the difference between the training semantics of the sample identifier and the semantics of the sample category, the model parameters of the image generation model are updated, and the image generation model is trained, which can improve the similarity between the training semantics of the sample identifier and the semantics of the sample category, that is, the similarity between the sample characteristics of the sample image under the sample category and the semantics of the sample category can be improved, which is conducive to reducing the overfitting degree of the image generation model and improving the training effect of the image generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 is a schematic diagram of the principle of an image generation model provided in an embodiment of the present application;

[0023] FIG2 is a schematic diagram of the architecture of an image generation system provided in an embodiment of the present application;

[0024] FIG3 is a flow chart of an image processing method provided in an embodiment of the present application;

[0025] FIG4 is a schematic diagram of the structure of an image generation model in the training phase provided by an embodiment of the present application;

[0026] FIG5 is a schematic diagram of the structure of an image generation model in which a training module is added during a training phase according to an embodiment of the present application;

[0027] FIG6 is a schematic diagram of a process for determining initial characterization information of a sample identifier provided by an embodiment of the present application;

[0028] FIG7 is a flow chart of another image processing method provided in an embodiment of the present application;

[0029] FIG8 is a schematic diagram of the structure of a trained image generation model provided in an embodiment of the present application;

[0030] FIG9 is a schematic diagram of the structure of another trained image generation model provided in an embodiment of the present application;

[0031] FIG10 is a comparative schematic diagram of an image generation effect provided by an embodiment of the present application;

[0032] FIG11 is a comparative schematic diagram of another image generation effect provided by an embodiment of the present application;

[0033] FIG12 is a schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0034] FIG13 is a schematic structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to more clearly understand the technical solutions provided by the embodiments of the present application, the key terms involved in the embodiments of the present application are first introduced here:

[0036] The embodiments of the present application relate to an image generation model. An image generation model, which can also be called a diffusion model, refers to a model that can generate a diffusion image semantically related to the text description information based on the sample features of the sample image under the sample category, given a sample image, the sample category (also called the concept category) to which the sample image belongs, and text description information. More specifically:

[0037] As shown in FIG1 , the training principle of the image generation model 101 can be roughly described as follows: a sample image 102 (for example, an image containing a hamster in FIG1 ), a sample category 103 to which the sample image 102 belongs (for example, the sample category “hamster” in FIG1 , or the sample category “hamster” in English), and a sample identifier 104 set for the sample image 102 (for example, the sample identifier “S*” in FIG1 ) are used as inputs to the image generation model 101. The training goal of the image generation model 101 is to enhance the similarity between the semantics (also referred to as an embedded concept) of the sample identifier 104 extracted by the image generation model 101 and the sample features of the sample image 102 under the sample category 103; for example, in FIG1 , the training goal of the image generation model 101 is to enhance the similarity between the semantics of “S*” extracted by the image generation model 101 and the features of “hamster” in the sample image 102. After the training of the image generation model 101 is completed, the semantics of the sample identification 104 extracted by the trained image generation model 101 can more accurately express the sample characteristics of the sample image 102 under the sample category 103 .

[0038] As shown in Figure 1, the application method of the image generation model 101 can be roughly described as follows: the input of the trained image generation model 101 is text description information 105 containing a sample identifier 104 (for example, "an S* in a bucket" in Figure 1, or "AS*in the bucket" in English). The trained image generation model 101 can diffuse based on the semantics of the sample identifier 104 (that is, the sample features of the sample image 102 under the sample category 103) to generate a diffused image 106 corresponding to the sample image 102. The generated diffused image 106 can include the sample features of the sample image 102 under the sample category 103, and can also be semantically related to other descriptive information in the text description information 105 except the sample identifier 104 (for example, "one", "in the bucket", and other descriptive information).

[0039] It should be noted that the image generation model involved in the embodiments of the present application is a model with basic image generation capabilities. The image generation model with basic image generation capabilities means that: based on given descriptive information, the image generation model has the ability to generate images that are semantically related to the given descriptive information; for example, if the given descriptive information is "a hamster", the image generation model can generate an image containing a hamster. It can be further understood that the image generation model provided in the embodiments of the present application is a pre-trained image generation model, and the pre-training enables the image generation model to have basic image generation capabilities. The training of the image generation model in the embodiments of the present application refers to fine-tuning the image generation model, using the basic image generation capabilities that the image generation model already has, based on the sample features of the sample image under the sample category, combined with the text description information for diffusion, to generate a diffused image corresponding to the sample image.

[0040] It should also be noted that the image generation model involved in the embodiments of the present application can be a machine learning model related to the field of computer vision technology of artificial intelligence.

[0041] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in ways similar to human intelligence.

[0042] Computer vision (CV) technology is the study of how machines can "see." Specifically, it refers to the use of cameras and computers to replace the human eye in identifying and measuring objects, and further processing the images to create images more suitable for human observation or transmission to instrumentation. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data.

[0043] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning. Pretrained models are the latest development in deep learning, integrating these techniques.

[0044] Furthermore, the embodiments of the present application do not limit the type of image generation model. The image generation model is a model with basic image generation capabilities. The image generation model may include but is not limited to any of the following: Diffusion Probabilistic Model (Diffusion Probabilistic Model, which can be referred to as Diffusion Model), Denoising Diffusion Probabilistic Model, and Stable-Diffusion Model.

[0045] In the training process of the image generation model, there is often a lack of multiple sample images under the same category. A single sample image is generally used to train the image generation model. However, the use of a single sample image to train the image generation model often leads to the problem of overfitting, resulting in poor training effect of the image generation model. Based on this, the embodiment of the present application proposes an image processing method that can reduce the overfitting degree of the image generation model and improve the training effect of the image generation model. Specifically, the image processing method proposed in the embodiment of the present application can train the image generation model based on the difference between the training semantics of the sample identifier (the training semantics of the sample identifier refers to the semantics of the sample identifier during the training process of the image generation model) and the semantics of the sample category. The similarity between the training semantics of the sample identifier and the semantics of the sample category can be improved. The training semantics of the sample identifier can be used to express the sample features of the sample image under the sample category. That is to say, the similarity between the sample features of the sample image under the sample category and the semantics of the sample category can be improved. This is conducive to reducing the overfitting degree of the image generation model and improving the training effect of the image generation model. In addition, the image generation model proposed in the embodiment of the present application can also initialize the training representation information of the sample identifier. The training representation information of the sample identifier refers to the representation information of the sample identifier extracted during the training process of the image generation model. Initializing the training representation information of the sample identifier refers to finding the initial representation information of the sample identifier that is close to the semantics of the sample image and the semantics of the sample category to which the sample image belongs, as the training starting point of the image generation model. This can quickly guide the image generation model in the appropriate training direction, improve the fitting efficiency of the image generation model, and improve the generalization of the image generation model.

[0046] As shown in FIG2 , the image processing system may include a model training device 201 and a model application device 202 . The model training device 201 and the model application device 202 may establish a direct communication connection via wired communication, or an indirect communication connection may be established via wireless communication. Specifically:

[0047] The model training device 201 can be used to train an image generation model. Specifically, a pre-trained image generation model is deployed in the model training device 201. The model training device 201 can train the image generation model (i.e., fine-tune the image generation model) based on sample data (the sample data may include a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image), so that the training semantics of the sample identifier extracted by the trained image generation model can more accurately express the sample features of the sample image under the sample category, thereby enabling the trained image generation model to have the ability to perform image diffusion based on the sample features of the sample image under the sample category.

[0048] The model application device 202 can be used to call a trained image generation model to perform image diffusion. Specifically, the trained image generation model can be deployed in the model application device 202. The model application device 202 can generate a diffused image corresponding to the sample image based on the given text description information (the text description information includes the sample identifier) ​​and the training semantics of the sample identifier (i.e., the sample features of the sample image under the sample category) as the diffusion basis.

[0049] The model training device 201 can be a terminal or a server, and the model application device 202 can be a terminal or a server. The terminal mentioned in the embodiment of the present application may include any of the following: a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device, a smart watch, a vehicle-mounted terminal, a smart home appliance, and an aircraft, etc., but is not limited to this. The server mentioned in the embodiment of the present application can be a separate physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. The embodiment of the present application does not limit this.

[0050] It should be noted that the image processing system shown in FIG2 is introduced by taking the model training device 201 and the model application device 202 as an example and being set up separately. In actual application scenarios, the model training device 201 and the model application device 202 in the image processing system can be integrated into different devices as shown in FIG2 , or the model training device 201 and the model application device 202 in the image processing system can be integrated into the same device, for example, the model training device 201 and the model application device 202 can be integrated into the same terminal or the same server. When the model training device 201 and the model application device 202 in the image processing system are integrated into the same device, the integrated device of the model training device 201 and the model application device 202 can be used to train the image generation model and can also be used to call the trained image generation model for image diffusion.

[0051] It can be understood that the image processing system shown in Figure 2 is for the purpose of more clearly illustrating the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems.

[0052] An embodiment of the present application provides an image processing method, which mainly introduces the training process of an image generation model (including: the structure of the image generation model, the training representation information of the initialized sample identification, and the loss information formed by the difference between the training semantics based on the sample identification and the semantics of the sample category, etc.). The image processing method can be executed by a computer device, which can be, for example, the model training device 201 in the image processing system shown in Figure 2 above, or it can be an integrated device of the model training device 201 and the model application device 202. As shown in Figure 3, the image processing method can include but is not limited to the following steps S301-S304:

[0053] S301, obtaining sample data for training an image generation model, where the sample data includes a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image.

[0054] The sample data used to train an image generation model may include: a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image. A sample image is a single image used to train the image generation model. The sample category to which the sample image belongs refers to the category to which the sample image belongs. The sample category to which the sample image belongs can be set based on the task type of the image generation task. If the task type of the image generation task is to perform image diffusion based on the sample image as a whole, then the sample category to which the sample image belongs can be the category to which the sample image as a whole belongs, for example, "landscape painting," "oil painting," etc. If the task type of the image generation task is to perform image diffusion based on sample objects within the sample image, then the sample category to which the sample image belongs can be the sample category to which the sample objects within the sample image belong, for example, "dog," "hamster," etc. The purpose of setting a sample identifier for the sample image is to ensure that the training semantics of the sample identifier extracted by the trained image generation model can more accurately express the sample characteristics of the sample image within the sample category. This allows, during the application phase of the trained image generation model, to achieve a better diffusion effect for the sample image based on the textual description information containing the sample identifier.

[0055] In the embodiments of the present application, steps S302 to S304 introduce the training process of the image generation model. Here, the structure of the image generation model in the model training phase is first introduced in conjunction with FIG4 , so that when describing the training process of the image generation model in steps S302 to S304, the structure of the image generation model shown in FIG4 can be referred to. As shown in FIG4 , the image generation model may include at least an image compression encoder 401, a first text encoder 402, a second text encoder 403, and an image diffusion sub-model 404. Among them:

[0056] (1) Image compression encoder 401: The image compression encoder can be used to convert an image (e.g., a sample image) from a pixel space to a coding space (the coding space can also be called a latent space). Converting an image from a pixel space to a coding space means performing coding processing on the image to obtain coding information of the sample image. The pixel space refers to the range composed of the image represented in the form of pixels. The coding space refers to the range composed of the coding information of the image. The coding process refers to compression processing. The coding information refers to the compressed information of the image. In addition, the sample image can be subjected to noise processing in the coding space. Specifically, after the image compression encoder converts the sample image from the pixel space to the coding space and obtains the coding information of the sample image, the coding information of the sample image can be subjected to noise processing to obtain a noisy image corresponding to the sample image. Noising processing refers to adding noise.

[0057] The embodiment of the present application does not limit the type of image compression encoder. The embodiment of the present application is described by taking the image compression encoder as a compression encoder (Encoder) in a VAE (Variational Autoencoder) as an example.

[0058] (2) First Text Encoder (First Text Encoder) 402: The first text encoder can be used to extract training representation information of the sample identifier, and the training representation information of the sample identifier can be used to represent the training semantics of the sample identifier. This embodiment of the application does not limit the type of the first text encoder. This embodiment of the application uses the example of the first text encoder being a text encoder in the CLIP model for illustration.

[0059] (3) Second Text Encoder (Second Text Encoder) 403: The second text encoder can be used to extract representation information of the sample category, and the representation information of the sample category can be used to represent the semantics of the sample category. This embodiment of the application does not limit the type of the second text encoder. This embodiment of the application uses the example of the second text encoder being a text encoder in the CLIP model for illustration.

[0060] (4) Image Diffusion Sub-Model (UNet) 404: During the training phase of the image generation model, the image diffusion sub-model can be used to predict the noise added to the noise image corresponding to the sample image based on the training representation information of the sample identification, and obtain the encoding information of the predicted noise.

[0061] Based on the relevant introduction to the image generation model structure in the embodiment shown in Figure 4, the general training principle of the image generation model is as follows: ① Call the image compression encoder 401 to convert the sample image from the pixel space to the coding space to obtain the coding information of the sample image. The coding information of the sample image can be denoised in the coding space to obtain a noisy image corresponding to the sample image. The noise added by the denoising process is the label noise; ② Call the first text encoder 402 to extract the training representation information of the sample identifier. The training representation information of the sample identifier can be used to represent the training semantics of the sample identifier. Call the second text encoder 403 to extract the training representation information of the sample category. The representation information of the sample category can be used to identify the semantics of the sample category; ③ Call the image diffusion sub-model 404 to predict the noise added to the noise image corresponding to the sample image based on the training representation information of the sample identifier to obtain the predicted noise; ④ Based on the difference between the predicted noise and the label noise, and the difference between the training representation information of the sample identifier and the representation information of the sample category (i.e., the difference between the training semantics of the sample identifier and the semantics of the sample category), update the parameters of the image generation model to train the image generation model. The above ①-④ describe any training process of the image generation model. The image generation model can be iteratively trained according to the above ①-④ until the training termination condition is met to obtain a trained image generation model. Among them, meeting the training termination condition may include any of the following: the number of iterative training reaches a threshold number (the threshold number can be set based on experience and used to control the number of training times, for example, the threshold number is 100 times), and the loss information of the image generation model is less than the loss threshold (the loss threshold can be set based on experience and used to control the loss information of the image generation model).

[0062] After introducing the structure of the image generation model and the general training principle of the image generation model in conjunction with FIG4 , the training process of the image generation model is described in detail below in conjunction with steps S302 to S304 .

[0063] S302: Calling an image generation model to extract training representation information of sample identification.

[0064] In step S302, the image generation model is called to extract the training representation information of the sample identifier, specifically, the text encoder in the image generation model is called (the text encoder here specifically refers to the first text encoder) to extract the training representation information of the sample identifier. The training representation information of the sample identifier can be used to represent the training semantics of the sample identifier, and the training semantics of the sample identifier can be used to express the sample characteristics of the sample image under the sample category.

[0065] Specifically, as shown in Figure 4, the text encoder may include a tokenizer layer, an embedding layer, and a text transformer layer. The tokenizer layer may be called to perform word segmentation on the sample identifier to obtain one or more word segments contained in the sample identifier. The tokenizer layer may be called to map each word segment to a token corresponding to each word segment. A token is a number that can be recognized by a computer device. The embedding layer may be called to convert the token of each word segment into a corresponding embedding vector. The transformer layer may be called to perform semantic understanding on the embedding vectors of each word segment to obtain training representation information of the sample identifier.

[0066] S303 , calling the image generation model based on the training representation information of the sample identifier, predicting the noise added to the noise image corresponding to the sample image, and obtaining the predicted noise.

[0067] In step S303, the image generation model is called to predict the noise to be added to the noise image corresponding to the sample image based on the training representation information of the sample identifier to obtain the predicted noise. Specifically, the image diffusion submodel in the image generation model is called to predict the noise to be added to the noise image corresponding to the sample image based on the training representation information of the sample identifier to obtain the predicted noise. As described above, the noise image corresponding to the sample image is obtained by converting the sample image from pixel space to coding space and then performing noise processing on the coding information of the sample image in the coding space. The process of predicting noise also occurs in the coding space. That is, the image generation model (specifically, the image diffusion submodel in the image generation model) can be called in the coding space to predict the noise to be added to the noise image based on the training representation information of the sample identifier to obtain the predicted noise. During the training process of the image processing model in the embodiment of the present application, the sample image is converted from the pixel space to the coding space, the sample image is denoised in the coding space, and the noise is predicted for the noisy image corresponding to the sample image in the coding space. This is because the dimension of the pixel space is higher than that of the coding space. Noising the sample image in the coding space and predicting the noise for the noisy image corresponding to the sample image in the coding space can reduce the amount of data processed during the denoising process and the noise prediction, and can improve the efficiency of the denoising process and the noise prediction, thereby improving the training efficiency of the image generation model to a certain extent.

[0068] Furthermore, the image diffusion sub-model may include N cross-attention modules, any one of the N cross-attention modules may be represented as the i-th cross-attention module, N is an integer greater than or equal to 2, and i is a positive integer less than or equal to N; in the encoding space, calling the image generation model (specifically the image diffusion sub-model in the image generation model) to predict the noise added to the noise image based on the training representation information of the sample identifier to obtain the process of predicting the noise may specifically include: when i=1, calling the i-th cross-attention module to perform correlation calculation on the training representation information of the sample identifier and the noise image to obtain the cross-attention result of the i-th cross-attention module. The correlation calculation here refers to calculating the similarity between the training representation information of the sample identifier and the noise image, and taking the similarity as the cross-attention result of the i-th cross-attention module; when i>1, calling the i-th cross-attention module to perform correlation calculation on the cross-attention result of the i-1-th cross-attention module and the training representation information of the sample identifier, and obtaining the cross-attention result of the i-th cross-attention module; the cross-attention result of the N-th cross-attention module is determined as the predicted noise, and the correlation calculation here refers to calculating the similarity between the cross-attention result of the i-1-th cross-attention module and the training representation information of the sample identifier, and taking the similarity as the cross-attention result of the i-th cross-attention module.

[0069] S304 : updating the model parameters of the image generation model according to the difference between the labeled noise and the predicted noise, and the difference between the training semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model.

[0070] After the image generation model is called based on the training representation information of the sample identifier to predict the noise added to the noise image corresponding to the sample image, and the predicted noise is obtained, the model parameters of the image generation model can be updated based on the difference between the labeled noise and the predicted noise, and the difference between the training semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model. Specifically, the process of updating the model parameters of the image generation model based on the difference between the labeled noise and the predicted noise, and the difference between the training semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model, can specifically include:

[0071] (1) The first loss information can be determined based on the difference between the marked noise and the predicted noise. The first loss information can be specifically referred to in the following formula 1: L LDM =E ε(x) ,y,∈~N(0,1),t[||∈-∈ θ (z t ,t,c)|| 2] Formula 1

[0072] In the above formula 1: L LDM represents the first loss information; x represents the sample image; c represents the training representation information of the sample identification; t represents the time step; ∈ represents the label noise; ∈ θ represents the prediction noise; z t It represents the noise image corresponding to the sample image obtained by performing noise processing on the coding information of the sample image in the coding space at time step t.

[0073] (2) The second loss information can be determined based on the difference between the training representation information of the sample identifier and the representation information of the sample category (i.e., the difference between the training semantics of the sample identifier and the semantics of the sample category). The second loss information can be specifically referred to in the following formula 2:

[0074] In the above formula 2: L CL represents the second loss information; T represents a text encoder (including the first text encoder or the second text encoder); c p represents the sample identification, T(c p ) represents the training representation information of the sample identification extracted by calling the text encoder (the text encoder here specifically refers to the first text encoder); c c represents the sample category, T(c c ) represents the representation information of the sample category extracted by calling the text encoder (the text encoder here specifically refers to the second text encoder); α cl Represents a weight parameter. The embodiment of the present application does not limit the specific parameter value of the weight parameter. The specific parameter value of the weight parameter can be set based on an empirical value. For example, the parameter value of the weight parameter is set to 1.

[0075] The representation information of the sample category can be extracted by calling the text encoder in the image generation model (the text encoder here specifically refers to the second text encoder). The process of the second text encoder extracting the representation information of the sample category is similar to the process of the first text encoder extracting the training representation information of the sample identifier in the above step S302. For details, please refer to the process of the first text encoder extracting the training representation information of the sample identifier in the above step S302, which will not be repeated here; the representation information of the sample category can be used to represent the semantics of the sample category.

[0076] (3) The first loss information and the second loss information can be weighted and summed to obtain the loss information of the image generation model. The embodiment of the present application does not limit the specific weight values ​​of the first weight used to weight the first loss information and the second weight used to weight the second loss information. The specific weight values ​​of the first weight and the second weight can be determined based on empirical values. For example, the weight values ​​of the first weight and the second weight can be set to 1. For another example, the weight value of the first weight can be set to 1, and the weight value of the second weight can be set to 0.5, and so on. The model parameters of the image generation model can be updated in the direction of minimizing the loss information of the image generation model to train the image generation model.

[0077] Furthermore, when updating the model parameters of the image generation model in the direction of minimizing the loss information of the image generation model, there can be two ways to update the model parameters of the image generation model: the first updating method is to directly update the model parameters of the image generation model, that is, the original model parameters of the image generation model can be updated in the direction of minimizing the loss information of the image generation model to train the image generation model; the second updating method is to add a training module to the image generation model, keep the original model parameters of the image generation model unchanged, and update the model parameters of the training module, that is, the original model parameters of the image generation model can be kept unchanged in the direction of minimizing the loss information of the image generation model, and update the model parameters of the training module to train the image generation model. The first updating method and the second updating method are introduced in detail below:

[0078] In the first updating method, the model parameters of the first text encoder and the image diffusion sub-model in the image generation model can be updated in a direction that minimizes the loss information of the image generation model, while the model parameters of the image compression encoder and the second text encoder remain unchanged, so as to train the image generation model. In other words, the essence of the image generation model training in this application is to train the first text encoder and the image diffusion sub-model in the image generation model.

[0079] In the second updating method, as shown in FIG5 , a training module (LoRA module) can be added to the image generation model. The training module can be added to the text encoder (the text encoder here specifically refers to the first text encoder) for extracting the training representation information of the sample identifier, and added to the image diffusion submodule; for the text encoder, the training module can be specifically added to the conversion layer of the text encoder. The training module can be used to reduce the dimension of the training representation information of the sample identifier output by the conversion layer, and then restore it to the original dimension of the training representation information of the sample identifier through dimensionality increase; for the image diffusion submodel, the image diffusion submodel can include multiple cross-attention modules. The training module can be added to the linear layer of each cross-attention module. The training module can be used to reduce the dimension of the result output by the linear layer, and then restore it to the original dimension of the result through dimensionality increase. Based on this, the original model parameters in the text encoder and the image diffusion submodel can be kept unchanged in the direction of minimizing the loss information of the image generation model, and the model parameters of the training modules added in the text encoder and the image diffusion submodel can be updated to train the image generation model. That is to say, the essence of the image generation model training in this application is to train the first text encoder of the image generation model and the training modules added to the image diffusion sub-model.

[0080] In step S304, the image generation model is trained by updating the model parameters of the image generation model based on the difference between the training semantics of the sample identifier and the semantics of the sample category. This can improve the similarity between the training semantics of the sample identifier and the semantics of the sample category, associate the training semantics of the sample identifier with the semantics of the sample category, and better utilize the prior knowledge of the image generation model (i.e., the semantics of the sample category). This can reduce the overfitting of the image generation model and improve the training effect of the image generation model. In the first updating method, directly updating the original model parameters of the image generation model can reduce the overfitting of the image generation model and improve the training effect of the image generation model. However, in general, the model parameters of the image generation model are relatively large, and directly updating the original model parameters of the image generation model may lose a certain amount of model training efficiency. In the second updating method, compared with the original model parameters of the image generation model, the model parameters of the training module are small. Updating the model parameters of the training module can reduce the overfitting of the image generation model, improve the training effect of the image generation model, and also improve a certain amount of model training efficiency.

[0081] It should be noted that in the above steps S301-S304, the training steps of extracting the training representation information of the sample identifier, predicting the noise added to the noisy image, and updating the model parameters of the image generation model can be used to train the image generation model once. The image generation model can be iteratively trained according to the above training steps until the training termination condition is reached to obtain a trained image generation model. The iterative training process of the image generation model uses the initial representation information of the sample identifier as the training starting point. The initial representation information of the sample identifier can be determined by minimizing the difference between the semantics of the sample image, the semantics of the sample category, and the native semantics of the sample identifier. The training starting point can specifically refer to: using the initial representation information of the sample identifier as the training representation information of the sample identifier to participate in the first training process of the image generation model. It can be seen that by minimizing the differences between the semantics of the sample image, the semantics of the sample category and the native semantics of the sample identifier, the training semantics of the sample identifier represented by the initial representation information of the determined sample identifier can be close to the semantics of the sample image and the semantics of the sample category. The initial representation information of the sample identifier is used to participate in the first training of the image generation model, which can quickly guide the image generation model in the appropriate training direction, improve the fitting efficiency of the image generation model, and improve the generalization of the image generation model.

[0082] Furthermore, the semantics of the sample image can be represented by the representation information of the sample image, and the representation information of the sample image can be obtained by semantically understanding the sample image using an image semantic encoder. The embodiment of the present application does not limit the type of the image semantic encoder. The embodiment of the present application takes the image semantic encoder as an example of the Image Encoder in the CLIP (Contrastive Language-Image Pre-Training) model. The image semantic encoder can be a trained image encoder that already has semantic understanding capabilities; the semantics of the sample category can be represented by the representation information of the sample category, and the representation information of the sample category can be extracted by the second text encoder of the image generation model; the native semantics of the sample identifier refers to the semantics of the sample identifier before the image generation model is trained. The native semantics of the sample identifier can be represented by the native representation information of the sample identifier, and the native representation information of the sample identifier can be extracted for the first time using the first text encoder of the image generation model. The first extraction refers to the extraction using the first text encoder before training, that is, the native representation information is in the image generation model. The representation information of the sample identifier is extracted before training; based on this, the process of determining the initial representation information of the sample identifier by minimizing the difference between the semantics of the sample image, the semantics of the sample category and the native semantics of the sample identifier can specifically include: extracting the representation information of the sample image, which can be used to represent the semantics of the sample image; extracting the representation information of the sample category, which can be used to represent the semantics of the sample category; averaging the representation information of the sample image and the representation information of the sample category to obtain average representation information; extracting the native representation information of the sample identifier, and determining the initial representation information of the sample identifier by minimizing the distance between the native representation information of the sample identifier and the average representation information.

[0083] Optionally, the representation information of the sample image may include the original image representation information of the sample image and the background-removed representation information of the background-removed image corresponding to the sample image. Specifically, the original image representation information of the sample image may be extracted, and the original image representation information of the sample image may be used to represent the original image semantics of the sample image, and the original image representation information of the sample image may be obtained by performing semantic understanding on the sample image using an image semantic encoder; the background-removed representation information of the background-removed image corresponding to the sample image may be extracted, and the background-removed representation information may be used to represent the semantics of the background-removed image, and the background-removed image may be obtained by removing the background from the sample image, and the background-removed representation information may be obtained by performing semantic understanding on the background-removed image using an image semantic encoder; the original image representation information and the background-removed representation information may be determined as the representation information of the sample image. Furthermore, as shown in FIG6 , the original image representation information of the sample image (for example, the image containing the hamster and the background in FIG6 ), the background-removed representation information of the background-removed image corresponding to the sample image (for example, the image containing the hamster but not the background in FIG6 ), and the representation information of the sample category (for example, the sample category “hamster” in FIG6 ) can be averaged to obtain the average representation information, and the native representation information of the sample identifier (for example, the sample identifier “S*” in FIG6 ) can be extracted. By minimizing the distance between the native representation information of the sample identifier and the average representation information, the initial representation information of the sample identifier is determined. For details, see the following formula 3:

[0084] In the above formula 3, L PE represents the initial representation information of the sample identifier; T represents the text encoder (including the first text encoder or the second text encoder); c p represents the sample identification, T(c p ) represents the native representation information of the sample identifier extracted by calling the text encoder (the text encoder here specifically refers to the first text encoder); c c represents the sample category, T(c c ) represents the representation information of the sample category extracted by calling the text encoder (the text encoder here specifically refers to the second text encoder); x represents the sample image, x m represents the background-removed image corresponding to the sample image; I represents the image semantic encoder, I(x) represents the representation information of the sample image obtained by calling the image semantic encoder to perform semantic understanding on the sample image, and I(x m ) indicates calling the image semantic encoder to perform semantic understanding on the background-removed image corresponding to the sample image, and obtaining the background-removed representation information of the background-removed image; θ m Indicates the rule or algorithm used for averaging; θ m (I(x),I(x m ),T(cc )) represents the average representation information.

[0085] It is easy to see that when determining the initial representation information of the sample identifier, the method of not introducing the background-removed image corresponding to the sample image is more focused on the sample image as a whole, and this method is more suitable for image generation tasks that perform image diffusion based on the sample image as a whole. When determining the initial representation information of the sample identifier, the method of introducing the background-removed image corresponding to the sample image is more focused on not only the sample image as a whole, but also the sample objects in the sample image (usually, the background-removed image obtained after background removal processing of the sample image contains the sample objects in the sample image). This method is more suitable for image generation tasks that perform image diffusion based on the sample objects in the sample image. In other words, the initial representation information of the sample identifier can be determined according to the determination method corresponding to the task type of the image generation task, so that the image generation model trained based on the initial representation information of the sample identifier can be more adapted to the image generation task.

[0086] It is worth noting that during the iterative training process of the image generation model, the noise images corresponding to the sample images used in each training process of the image generation model can be different; the different noise images corresponding to the sample images can specifically mean that the label noise added to the sample images during each training process of the image generation model is different. Among them, the different label noises can include any of the following: the type of label noise added in each training process of the image generation model is different (for example, the label noise added in the first training process is Gaussian noise, and the label noise added in the second training process is gamma noise...), the type of label noise added in each training process of the image generation model is the same but the intensity is different (for example, the label noise added in each training process is Gaussian noise, and the intensity of the Gaussian noise gradually increases with the increase in the number of training times). By using different noise images in each training process of the image generation model (i.e., using different label noises for noise addition), the image generation model can be trained to achieve better image generation effects for different noise images, which is conducive to improving the training effect of the image generation model.

[0087] In an embodiment of the present application, during the training process of the image generation model, by minimizing the difference between the semantics of the sample image, the semantics of the sample category and the native semantics of the sample identifier, the initial representation information of the sample identifier is determined for the first training process of the image generation model, so that the training semantics of the sample identifier represented by the determined initial representation information of the sample identifier can be close to the semantics of the sample image and the semantics of the sample category. Using the initial representation information of the sample identifier to participate in the first training of the image generation model can quickly guide the image generation model in the appropriate training direction, improve the fitting efficiency of the image generation model, and improve the generalization of the image generation model. Based on the difference between the training semantics of the sample identifier and the semantics of the sample category, the model parameters of the image generation model are updated to train the image generation model, which can improve the similarity between the training semantics of the sample identifier and the semantics of the sample category, and can associate the training semantics of the sample identifier with the semantics of the sample category, so as to better utilize the prior knowledge of the image generation model (i.e., the semantics of the sample category), thereby reducing the degree of overfitting of the image generation model and improving the training effect of the image generation model.

[0088] The embodiment of the present application provides an image processing method, which mainly introduces the application process of the trained image generation model (including: the image diffusion process based on text description information, and the specific application scenarios of the trained image generation model, etc.). The image processing method can be executed by a computer device, which can be, for example, an integrated device of a model training device 201 and a model application device 202, or the image processing method can be collaboratively executed by the model training device 201 and the model application device 202. In the embodiment of the present application, the devices that collaboratively execute the image processing method are collectively referred to as computer devices. As shown in Figure 7, the image processing method may include but is not limited to the following steps S701-step S707:

[0089] S701, obtaining sample data for training an image generation model, where the sample data includes a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image.

[0090] In the embodiment of the present application, the execution process of step S701 is the same as the execution process of step S301 in the embodiment shown in Figure 3 above. The specific execution process can be found in the relevant description of step S301 in the embodiment shown in Figure 3 above, and will not be repeated here.

[0091] S702: Calling an image generation model to extract training representation information of sample identification.

[0092] In the embodiment of the present application, the execution process of step S702 is the same as the execution process of step S302 in the embodiment shown in Figure 3 above. The specific execution process can be found in the relevant description of step S302 in the embodiment shown in Figure 3 above, and will not be repeated here.

[0093] S703 , calling the image generation model based on the training representation information of the sample identifier, predicting the noise added to the noise image corresponding to the sample image, and obtaining the predicted noise.

[0094] In the embodiment of the present application, the execution process of step S703 is the same as the execution process of step S303 in the embodiment shown in Figure 3 above. The specific execution process can be found in the relevant description of step S303 in the embodiment shown in Figure 3 above, and will not be repeated here.

[0095] S704, updating the model parameters of the image generation model according to the difference between the labeled noise and the predicted noise, and the difference between the training semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model, wherein the trained image generation model includes the trained text encoder and the trained image diffusion sub-model.

[0096] In the embodiment of the present application, the execution process of step S704 is the same as the execution process of step S304 in the embodiment shown in Figure 3 above. The specific execution process can be found in the relevant description of step S304 in the embodiment shown in Figure 3 above, and will not be repeated here.

[0097] In the embodiment of the present application, steps S705 to S707 introduce the application process of the trained image generation model. Here, the structure of the trained image generation model is first introduced in conjunction with Figure 8, so that when introducing steps S705 to S707, it can be based on the structure of the trained image generation model. When introducing the application process of the trained image generation model in detailed steps S705 to S707, you can refer to the structure of the trained image processing model shown in Figure 8. The structure of the trained image generation model is similar to the structure of the image generation model in the model training stage shown in Figure 4. As shown in Figure 8, the trained image generation model can at least include an image compression encoder 401, a trained text encoder 801 (the trained text encoder mentioned here specifically refers to the trained first text encoder), a trained image diffusion sub-model 802, and an image compression decoder 803. Among them:

[0098] (1) Image compression encoder 401: The image compression encoder can be used to convert a sample image from a pixel space to a coding space to obtain coding information of the sample image. Furthermore, the sample image can be subjected to noise processing in the coding space. Specifically, after the image semantic encoder converts the sample image from a pixel space to a coding space to obtain coding information of the sample image, the coding information of the sample image can be subjected to noise processing to obtain a noisy image corresponding to the sample image.

[0099] (2) Trained first text encoder 801: The trained first text encoder can be used to extract representation information of text description information. The text description information may include sample identifiers. The representation information of the text description information can be used to represent the semantics of the text description information.

[0100] (3) Trained image diffusion sub-model 802: In the application stage of the trained image generation model, the trained image diffusion sub-model can be used to perform denoising on the noise image corresponding to the sample image based on the representation information of the text description information to obtain the encoding information of the diffusion image corresponding to the sample image; denoising refers to removing noise from the noisy image.

[0101] (4) Image Compression Decoder 803: The decoding process of the image compression decoder is the inverse of the encoding process of the image compression encoder. During the application phase of the trained image generation model, the image compression decoder can be used to convert the encoded information of the diffusion image from the code space to the pixel space to obtain the diffusion image. Converting the encoded information of the diffusion image from the code space to the pixel space refers to decoding the encoded information of the diffusion image, and decoding refers to decompression.

[0102] The embodiment of the present application does not limit the type of image compression decoder. The embodiment of the present application is described by taking the image compression decoder as a compression decoder (Decoder) in a VAE (Variational Autoencoder) as an example.

[0103] Corresponding to the training stage of the image generation model, if the image generation model is not added with a training module during the training process, that is, when the model parameters are updated, the original model parameters of the image generation model are directly updated, the structure of the trained image generation model is shown in Figure 8; if the image generation model is added with a training module during the training process, that is, when the model parameters are updated, the original model parameters of the image generation model are kept unchanged, and the model parameters of the added training module are updated, the structure of the trained image generation model is shown in Figure 9; the difference between Figure 8 and Figure 9 is that in the structure of the trained image generation model shown in Figure 9, the trained text encoder (the trained text encoder here specifically refers to the trained first text encoder) encoder) and the trained image diffusion sub-model; corresponding to the training stage of the image generation model, the trained training module can be added to the conversion layer of the trained text encoder, and the trained training module can be used to perform dimensionality reduction processing on the representation information of the text description information output by the conversion layer, and then restore it to the original dimension of the representation information of the text description information through dimensionality increase processing; the trained image diffusion sub-model may include multiple cross-attention modules, and the trained training module can be added to the linear layer of each cross-attention module, and the trained training module can be used to perform dimensionality reduction processing on the result output by the linear layer, and then restore it to the original dimension of the result through dimensionality increase processing.

[0104] Based on the above introduction to the structure of the trained image generation model, the general application process of the trained image generation model can be: ① Call the image compression encoder 401 to convert the sample image from the pixel space to the coding space to obtain the coding information of the sample image. The sample image can be denoised in the coding space to obtain the noisy image corresponding to the sample image; ② Call the trained first text encoder 801 to extract the representation information of the text description information. The text description information contains the sample identifier, and the representation information of the text description information can be used to represent the semantics of the text description information; ③ The trained image diffusion sub-model 802 denoises the noise image corresponding to the sample image based on the representation information of the text description information to obtain the coding information of the diffusion image corresponding to the sample image; ④ Call the image compression decoder 803 to convert the coding information of the diffusion image from the coding space to the pixel space to obtain the diffusion image. The diffusion image is semantically related to the text description information. The semantic correlation here means that the image content of the diffusion image can express the semantics of the text description information.

[0105] After introducing the structure of the trained image generation model and the application process of the trained image generation model in conjunction with FIG8 and FIG9 , the application process of the trained image generation model is described in detail below in conjunction with steps S705 to S707 .

[0106] S705: Obtain text description information including the sample identifier.

[0107] S706: Call the trained text encoder to extract representation information of the text description information.

[0108] In step S706, the process of calling the trained text encoder to extract the representation information of the text description information is similar to the process of calling the first text encoder to extract the training representation information of the sample identifier in the embodiment shown in Figure 3 above. Specifically, the trained text encoder may include a token layer, an embedding layer, and a conversion layer. The token layer may be called to perform word segmentation processing on the text description information to obtain one or more word segments contained in the text description information. The token layer may be called to map each word segment to a token corresponding to each word segment. The token is a number that can be recognized by a computer device. The embedding layer may be called to convert the token of each word segment into a corresponding embedding vector. The conversion layer may be called to perform semantic understanding on the embedding vectors of each word segment to obtain the representation information of the text description information.

[0109] S707 , calling the trained image diffusion sub-model, performing denoising on the noisy image based on the representation information of the text description information, and generating a diffusion image corresponding to the sample image.

[0110] As described above, the noise image corresponding to the sample image is obtained by converting the sample image from the pixel space to the coding space and then performing noise processing on the coding information of the sample image in the coding space. The denoising process also occurs in the coding space. That is, in the coding space, the trained image diffusion sub-model can be called to perform denoising on the noise image based on the representation information of the text description information to obtain the coding information of the diffusion image corresponding to the sample image. The image compression decoder can be called to convert the coding information of the diffusion image from the coding space to the pixel space to obtain the diffusion image. The diffusion image is semantically related to the text description information. In the application process of the trained image processing model in the embodiment of the present application, the sample image is converted from the pixel space to the coding space, the sample image is denoised in the coding space, and the noise image corresponding to the sample image is denoised in the coding space. This is because the dimension of the pixel space is higher than that of the coding space. The sample image is denoised in the coding space and the noise image corresponding to the sample image is denoised in the coding space. This can reduce the amount of data processed during the denoising and denoising processes, improve the efficiency of the denoising and denoising processes, and thus improve the image generation efficiency to a certain extent.

[0111] In step S707, in the coding space, the trained image diffusion sub-model is called to perform denoising on the noisy image based on the representation information of the text description information, and the process of obtaining the coding information of the diffusion image corresponding to the sample image is similar to the process of predicting noise in the embodiment shown in FIG3 . Specifically, the trained image diffusion sub-model may include N cross-attention modules, any one of the N cross-attention modules may be represented as the i-th cross-attention module, N is an integer greater than or equal to 2, and i is a positive integer less than or equal to N; when i=1, the i-th cross-attention module is called to perform correlation calculation on the representation information of the text description information and the noise image, and obtain the cross-attention result of the i-th cross-attention module; when i>1, the i-th cross-attention module is called to perform correlation calculation on the cross-attention result of the i-1-th cross-attention module and the representation information of the text description information, and obtain the cross-attention result of the i-th cross-attention module; the cross-attention result of the N-th cross-attention module is determined as the coding information of the diffusion image.

[0112] The above steps S705 to S707 have detailed the application process of the trained image generation model. Based on this, we will introduce two types of application scenarios of the trained image generation model. The two types of application scenarios correspond to the two types of image generation tasks. Specifically:

[0113] If the task type of the image generation task is to diffuse the image based on the sample image as a whole, then the sample category to which the sample image belongs can be the category to which the sample image as a whole belongs; when the sample category is the category to which the sample image as a whole belongs, the sample features of the sample image under the sample category can be the overall image features of the sample image under the sample category, the training semantics of the sample identifier can be used to express the overall image features of the sample image under the sample category, and the diffused image corresponding to the sample image can be an image diffused and generated based on the sample image as a whole. For example, in an image enhancement scenario, the goal of the image enhancement scenario is to diffuse a high-quality (e.g., higher clarity, fuller image, etc.) sample image based on the sample image. In this case, the sample category can be the category to which the sample image as a whole belongs (e.g., the sample category is "landscape image"), and the text description information containing the sample identifier can be, for example, "a high-quality S*". Based on this, a high-quality sample image can be diffused based on the sample image, which is conducive to rapid image enhancement. For another example, in an image restoration scenario, the goal is to diffuse restored sample images based on a sample image. In this case, the sample category can be the category to which the sample image as a whole belongs (for example, the sample category is "landscape image"), and the text description information containing the sample identifier can be, for example, "a restored S*". Based on this, the restored sample image can be diffused based on the sample image, which is conducive to the rapid realization of image restoration. For another example, in an image art creation scenario, the goal of image art creation is to diffuse images that are similar in style to the sample image but have innovative features based on the sample image. In this case, the sample category can be the category to which the sample image as a whole belongs (for example, the sample category is "oil painting"), and the text description information containing the sample identifier can be, for example, "S* of mixed ink painting". Based on this, images that are similar in style to the sample image but have innovative features can be diffused based on the sample image, which helps artists explore new creative directions and forms of expression.

[0114] If the task type of an image generation task is to perform image diffusion based on a sample object in a sample image, then the sample category to which the sample image belongs can be the category to which the sample object in the sample image belongs; when the sample category is the category to which the sample object in the sample image belongs, the sample features of the sample image under the sample category can be the object features of the sample object in the sample image under the sample category; the training semantics of the sample identifier can be used to express the object features of the sample object in the sample image under the sample category; and the diffusion image corresponding to the sample image can be an image generated by diffusion based on the sample object in the sample image. For example, in a character design scenario, the goal of character design is to diffuse a virtual character corresponding to the character object based on the character object in the sample image. In this case, the sample category can be the category to which the sample object (i.e., the character object) in the sample image belongs (e.g., the sample category is "person"), and the textual description information containing the sample identifier can be, for example, "a cartoon S*," "a running cartoon S*," "a crying S*," etc. Based on this, a virtual character similar to the character object in the sample image but with different features, actions, expressions, and styles can be diffused based on the character object in the sample image, which has potential applications in role-playing or animation creation. For example, in a virtual fitting room scenario, the goal of the virtual fitting room is to diffuse person objects wearing different clothes based on the person objects in the sample image. In this case, the sample category can be the category to which the sample object (i.e., person object) in the sample image belongs (for example, the sample category is "person"), and the text description information containing the sample identifier can be, for example, "S* wearing a denim skirt", "S* wearing a suit", and "S* wearing Martin boots", etc. Based on this, person objects wearing different clothes can be diffused based on the person objects in the sample image, which can help preview the clothing effect and provide personalized shopping suggestions. For example, in the personalized product customization scenario, the goal of personalized products is to diffuse personalized products containing the target object based on the target object in the sample image. In this case, the sample category can be the category to which the animal object (e.g., Corgi) in the sample image belongs (e.g., the sample category is "dog"), and the text description information containing the sample identifier can be, for example, "cup containing S*", "mobile phone case containing S*", and "T-shirt containing S*", etc.; based on this, personalized products containing animal objects can be diffused based on the animal objects in the sample image.

[0115] In the embodiment of the present application, based on two optimization points in the training process of the image generation model (the first optimization point is that during the first training process of the image generation model, the initial representation information of the sample identifier is determined by minimizing the difference between the semantics of the sample image, the semantics of the sample category and the native semantics of the sample identifier, so that the training semantics of the sample identifier represented by the determined initial representation information of the sample identifier can be close to the semantics of the sample image and the semantics of the sample category, and the initial representation information of the sample identifier is used to participate in the first training of the image generation model, which can quickly guide the image generation model in the appropriate training direction, improve the fitting efficiency of the image generation model, and improve the image generation model. The generalization of the model; the second optimization point is to train the image generation model based on the difference between the training semantics of the sample identification and the semantics of the sample category, which can improve the similarity between the training semantics of the sample identification and the semantics of the sample category, and associate the training semantics of the sample identification with the semantics of the sample category, so as to better utilize the prior knowledge of the image generation model (that is, the semantics of the sample category), which can reduce the overfitting degree of the image generation model and improve the training effect of the image generation model), and can improve the ability to combine the sample features of the sample image under the sample category with the text semantics in the text description information, thereby improving the image generation effect of the trained image generation model.

[0116] The following is a comparison of the generation effects of the trained image generation model when the optimization points are introduced during the training process of the image generation model and when the optimization points are not introduced.

[0117] As shown in Figure 10, the first column of images are sample images, belonging to the sample category "hamster", the sample identifier set for the sample image is "[hamster*]", and the text description information containing the sample identifier is "[hamster*] is in the bucket"; the second column shows the generation effect of the trained image generation model when the first optimization point is not introduced during the training process of the image generation model; the third column shows the generation effect of the trained image generation model when the first optimization point is introduced during the training process of the image generation model. It can be seen that the image generation effect of the image generation model trained without the first optimization point is significantly worse than the image generation effect of the image generation model trained with the first optimization point. By introducing the first optimization point, the image generation model can be quickly guided in the appropriate training direction, improving the fitting efficiency of the image generation model and improving the generalization of the image generation model.

[0118] As shown in FIG. 11, the first column of images is the sample images. The sample category to which the sample images belong is "dog", the sample identifier set for the sample images is "[dog*]", and the text description information including the sample identifier is "An oil painting contains a [dog*] wearing a hat"; the second column is the generation effect of the trained image generation model when the second optimization point is not introduced during the training process of the image generation model; the third column is the generation effect of the trained image generation model when the second optimization point is introduced during the training process of the image generation model. It can be seen that the image generation effect of the trained image generation model without introducing the second optimization point is significantly worse than that of the trained image generation model when the second optimization point is introduced. By introducing the second optimization point, the similarity between the training semantics of the sample identifier and the semantics of the sample category can be improved, the training semantics of the sample identifier can be associated with the semantics of the sample category, and the prior knowledge of the image generation model (i.e., the semantics of the sample category) can be better utilized. This can reduce the overfitting degree of the image generation model, enabling the trained image processing model to better combine the sample features of the sample images under the sample category with the text semantics of the text description information, and the generated images are more natural and diverse.

[0119] Please refer to FIG. 12. FIG. 12 is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application. Please refer to FIG. 12. The image processing apparatus may include the following units:

[0120] An acquisition unit 1201, configured to acquire sample data for training an image generation model. The sample data includes sample images, the sample category to which the sample images belong, and the sample identifier set for the sample images;

[0121] A processing unit 1202, configured to call the image generation model to extract the training representation information of the sample identifier. The training representation information of the sample identifier is used to represent the training semantics of the sample identifier, and the training semantics of the sample identifier are used to express the sample features of the sample images under the sample category;

[0122] The processing unit 1202 is further configured to call the image generation model to predict the noise added to the noise image corresponding to the sample image based on the training representation information of the sample identifier, and obtain the predicted noise; the noise image is obtained by adding noise to the sample image, and the noise added by the noise addition process is labeled noise;

[0123] The processing unit 1202 is further configured to update the model parameters of the image generation model according to the difference between the labeled noise and the predicted noise, and the difference between the training semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model; the trained image generation model is used to perform denoising processing on the noise image based on the text description information including the sample identifier, and generate a diffusion image related to the semantics of the text description information.

[0124] In one implementation, the steps of extracting training representation information of sample identifiers, predicting noise added to a noisy image, and updating model parameters of the image generation model are used to train the image generation model once; the processing unit 1202 is further configured to perform the following steps:

[0125] According to the training steps, the image generation model is iteratively trained until the training termination condition is reached, and a trained image generation model is obtained;

[0126] The iterative training process of the image generation model takes the initial representation information of the sample identifier as the training starting point. The initial representation information of the sample identifier is determined by minimizing the difference between the semantics of the sample image, the semantics of the sample category and the native semantics of the sample identifier. The training starting point means: using the initial representation information of the sample identifier as the training representation information of the sample identifier to participate in the first training process of the image generation model.

[0127] In one implementation, during the iterative training of the image generation model, the noise images corresponding to the sample images used in each training process of the image generation model are different; the different noise images corresponding to the sample images refer to: different labeling noises added to the sample images during the noise processing in each training process of the image generation model;

[0128] The different labeling noises include any one of the following: different types of labeling noises added in each training process of the image generation model; and the same types of labeling noises added in each training process of the image generation model but different intensities.

[0129] In one implementation, the processing unit 1202 is configured to determine the initial representation information of the sample identifier by minimizing the difference between the semantics of the sample image, the semantics of the sample category, and the native semantics of the sample identifier, by performing the following steps:

[0130] Extracting representation information of the sample image, where the representation information of the sample image is used to represent the semantics of the sample image;

[0131] Extract the representation information of the sample category, which is used to represent the semantics of the sample category;

[0132] Averaging the representation information of the sample image and the representation information of the sample category to obtain average representation information;

[0133] Extracting native representation information of the sample identifier, where the native representation information of the sample identifier is used to represent the native semantics of the sample identifier;

[0134] The initial representation information of the sample identifier is determined by minimizing the distance between the native representation information of the sample identifier and the average representation information.

[0135] In one implementation, the processing unit 1202 is configured to extract the representation information of the sample image by performing the following steps:

[0136] Extracting original image representation information of the sample image, where the original image representation information is used to represent the original image semantics of the sample image;

[0137] Extracting background-removed representation information of the background-removed image corresponding to the sample image, where the background-removed representation information is used to represent the semantics of the background-removed image;

[0138] The original image representation information and the background removal representation information are determined as the representation information of the sample image.

[0139] In one implementation, the image generation model includes a text encoder and an image diffusion sub-model, wherein the text encoder is used to extract training representation information of sample identification, and the image diffusion sub-model is used to predict noise; a training module is added to the text encoder and the image diffusion sub-model;

[0140] The processing unit 1202 is configured to update the model parameters of the image generation model based on the difference between the labeled noise and the predicted noise, and the difference between the training semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model, and specifically to perform the following steps:

[0141] According to the difference between labeling noise and prediction noise, as well as the difference between the training semantics of sample identification and the semantics of sample category, the original model parameters in the text encoder and image diffusion sub-model are kept unchanged, and the model parameters of the training modules added in the text encoder and image diffusion sub-model are updated to train the image generation model.

[0142] In one implementation, predicting the noise added to the noisy image is performed by calling an image diffusion sub-model in the image generation model, where the image diffusion sub-model includes N cross-attention modules, and any one of the N cross-attention modules is denoted as an i-th cross-attention module, where N is an integer greater than or equal to 2, and i is a positive integer less than or equal to N.

[0143] The processing unit 1202 is configured to call the image generation model based on the training representation information of the sample identifier to predict the noise added to the noise image corresponding to the sample image. When the predicted noise is obtained, the processing unit 1202 is configured to perform the following steps:

[0144] When i=1, the i-th cross attention module is called to perform correlation calculation on the training representation information of the sample identifier and the noise image to obtain the cross attention result of the i-th cross attention module;

[0145] When i>1, the i-th cross attention module is called to perform correlation calculation on the cross attention result of the i-1-th cross attention module and the training representation information of the sample identifier to obtain the cross attention result of the i-th cross attention module;

[0146] The cross attention result of the Nth cross attention module is determined as the prediction noise.

[0147] In one implementation, the trained image generation model includes a trained text encoder and a trained image diffusion sub-model; the acquisition unit 1201 is further configured to perform the following steps:

[0148] Get text description information including sample identification;

[0149] The processing unit 1202 is further configured to perform the following steps:

[0150] Calling the trained text encoder to extract the representation information of the text description information; the representation information of the text description information is used to express the semantics of the text description information;

[0151] The trained image diffusion sub-model is called to denoise the noisy image based on the representation information of the text description information, and a diffusion image corresponding to the sample image is generated. The diffusion image is semantically related to the text description information.

[0152] In one implementation, the noise image is obtained by converting the sample image from pixel space to code space and then performing noise processing in the code space. The processing unit 1202 is configured to call the trained image diffusion sub-model to denoise the noise image based on the representation information of the text description information to generate a diffusion image corresponding to the sample image. Specifically, the processing unit 1202 is configured to perform the following steps:

[0153] In the coding space, calling the trained image generation model based on the representation information of the text description information, denoising the noisy image, and obtaining the coding information of the diffusion image;

[0154] The coding information of the diffusion image is converted from the coding space to the pixel space to obtain the diffusion image.

[0155] In one implementation, when the sample category is the category to which the entire sample image belongs, the sample features of the sample image under the sample category are the overall image features of the sample image under the sample category, the training semantics of the sample identifier is used to express the overall image features of the sample image under the sample category, and the diffusion image corresponding to the sample image is an image generated by diffusion based on the overall sample image.

[0156] When the sample category is the category to which the sample object in the sample image belongs, the sample features of the sample image under the sample category are the object features of the sample object in the sample image under the sample category. The training semantics of the sample identification are used to express the object features of the sample object in the sample image under the sample category. The diffusion image corresponding to the sample image is an image generated by diffusion based on the sample object in the sample image.

[0157] According to another embodiment of the present application, the various units in the image processing device shown in Figure 12 can be individually or all combined into one or several other units to constitute, or one (some) of the units can be further divided into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In actual applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the image processing device may also include other units. In actual applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0158] According to another embodiment of the present application, an image processing apparatus as shown in FIG12 can be constructed, and the image processing method of the embodiment of the present application can be implemented, by running a computer program capable of executing some or all of the steps involved in the method shown in FIG3 or FIG7 on a general-purpose computing device such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), a random access memory (RAM), and a read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable storage medium, and loaded into the above-mentioned computing device via the computer-readable storage medium and executed therein.

[0159] In an embodiment of the present application, in the process of training the image generation model using a single sample image, a sample identifier can be set for the sample image, and the training semantics of the sample identifier can be used to express the sample characteristics of the sample image under the sample category. Based on the difference between the training semantics of the sample identifier and the semantics of the sample category, the model parameters of the image generation model are updated, and the image generation model is trained. This can improve the similarity between the training semantics of the sample identifier and the semantics of the sample category. In other words, the similarity between the sample characteristics of the sample image under the sample category and the semantics of the sample category can be improved. This is conducive to reducing the overfitting degree of the image generation model and improving the training effect of the image generation model.

[0160] Based on the above method and device embodiments, an embodiment of the present application provides a computer device. Please refer to Figure 13, which is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device shown in Figure 13 includes at least a processor 1301, an input interface 1302, an output interface 1303, and a computer-readable storage medium 1304. The processor 1301, input interface 1302, output interface 1303, and computer-readable storage medium 1304 may be connected via a bus or other means.

[0161] Computer-readable storage medium 1304 may be stored in a memory of a computer device. Computer-readable storage medium 1304 is used to store a computer program, which includes computer instructions. Processor 1301 is used to execute the computer program stored in computer-readable storage medium 1304. Processor 1301 (or CPU (Central Processing Unit)) is the computing and control core of the computer device and is suitable for implementing computer programs, specifically loading and executing computer programs to implement corresponding method processes or corresponding functions.

[0162] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the operating system of the computer device. In addition, a computer program suitable for being loaded and executed by the processor is also stored in the storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (Non-Volatile Memory), such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0163] The computer device can, for example, be an integrated device of the model training device and the model application device in the image processing system shown in Figure 2. In a specific implementation, the processor 1301 can load and execute the computer program stored in the computer-readable storage medium 1304 to implement the corresponding steps in the above-mentioned image processing method shown in Figure 3 or Figure 7.

[0164] According to one aspect of the present application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the image processing methods provided in the various optional embodiments described above.

[0165] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: The method is performed by a computer device, and the method comprises: Acquire sample data for training an image generation model, the sample data including a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image; Calling the image generation model to extract training representation information of the sample identifier, where the training representation information of the sample identifier is used to represent the training semantics of the sample identifier, and the training semantics of the sample identifier is used to express sample features of the sample image under the sample category; The image generation model is called based on the training representation information of the sample identifier to predict the noise added to the noise image corresponding to the sample image to obtain the predicted noise; the noise image is obtained by performing noise processing on the sample image, and the noise added by the noise processing is the label noise; According to the difference between the labeled noise and the predicted noise, and the difference between the training semantics of the sample identifier and the semantics of the sample category, the model parameters of the image generation model are updated to train the image generation model; the trained image generation model is used to denoise the noisy image based on the text description information containing the sample identifier, so as to generate a diffusion image semantically related to the text description information.

2. The method according to claim 1, characterized in that The method further comprises: Iteratively training the image generation model until a training termination condition is reached, thereby obtaining a trained image generation model; The iterative training process of the image generation model takes the initial representation information of the sample identifier as the training starting point; wherein, the initial representation information of the sample identifier is determined by minimizing the difference between the semantics of the sample image, the semantics of the sample category and the native semantics of the sample identifier, and the training starting point means: using the initial representation information of the sample identifier as the training representation information of the sample identifier to participate in the first training process of the image generation model.

3. The method according to any one of claims 1 to 2, characterized in that: The process of determining initial representation information of the sample identifier by minimizing the difference between the semantics of the sample image, the semantics of the sample category and the native semantics of the sample identifier includes: Extracting representation information of the sample image, where the representation information of the sample image is used to represent the semantics of the sample image; Extracting representation information of the sample category, where the representation information of the sample category is used to represent the semantics of the sample category; Performing average processing on the representation information of the sample image and the representation information of the sample category to obtain average representation information; Extracting native representation information of the sample identifier, where the native representation information of the sample identifier is used to represent native semantics of the sample identifier; The initial characterization information of the sample identification is determined by minimizing the distance between the native characterization information of the sample identification and the average characterization information.

4. The method according to any one of claims 1 to 3, characterized in that: The extracting the representation information of the sample image includes: Extracting original image representation information of the sample image, where the original image representation information is used to represent the original image semantics of the sample image; Extracting background removal representation information of the background removal image corresponding to the sample image, wherein the background removal representation information is used to represent the semantics of the background removal image; The original image representation information and the background removal representation information are determined as the representation information of the sample image.

5. The method according to any one of claims 1 to 4, characterized in that: The image generation model includes a text encoder and an image diffusion sub-model, wherein the text encoder is used to extract training representation information of the sample identification, and the image diffusion sub-model is used to predict noise; a training module is added to the text encoder and the image diffusion sub-model; The difference between the label noise and the predicted noise, and the training semantics of the sample identifier and the sample category The method comprises: determining the difference between the semantics, updating the model parameters of the image generation model to train the image generation model, and comprising: According to the difference between the marked noise and the predicted noise, and the difference between the training semantics of the sample identifier and the semantics of the sample category, the original model parameters in the text encoder and the image diffusion sub-model are kept unchanged, and the model parameters of the training module added in the text encoder and the image diffusion sub-model are updated to train the image generation model.

6. The method according to any one of claims 1 to 5, characterized in that: Predicting the noise added to the noise image is performed by calling an image diffusion submodel in the image generation model, wherein the image diffusion submodel includes N cross-attention modules, and any one of the N cross-attention modules is represented as an i-th cross-attention module, where N is an integer greater than or equal to 2, and i is a positive integer less than or equal to N; The calling of the image generation model to predict the noise added to the noise image corresponding to the sample image based on the training representation information of the sample identifier to obtain the predicted noise includes: When i=1, calling the i-th cross attention module to perform correlation calculation on the training representation information of the sample identifier and the noise image, and obtaining the cross attention result of the i-th cross attention module; When i>1, the i-th cross attention module is called to perform correlation calculation on the cross attention result of the i-1-th cross attention module and the training representation information of the sample identifier to obtain the cross attention result of the i-th cross attention module; The cross-attention result of the Nth cross-attention module is determined as the predicted noise.

7. The method according to any one of claims 1 to 6, characterized in that: The trained image generation model includes a trained text encoder and a trained image diffusion sub-model; the method further includes: Obtaining text description information including the sample identifier; Calling the trained text encoder to extract representation information of the text description information; the representation information of the text description information is used to express the semantics of the text description information; The trained image diffusion sub-model is called to perform denoising on the noisy image based on the representation information of the text description information, so as to generate a diffusion image corresponding to the sample image, wherein the diffusion image is semantically related to the text description information.

8. The method according to any one of claims 1 to 7, characterized in that: The noise image is obtained by converting the sample image from a pixel space to a coding space and then performing a noise adding process in the coding space; calling the trained image diffusion sub-model to perform a denoising process on the noise image based on the representation information of the text description information to generate a diffusion image corresponding to the sample image, including: In the coding space, calling the trained image generation model based on the representation information of the text description information, denoising the noisy image, and obtaining the coding information of the diffusion image; The coding information of the diffusion image is converted from the coding space to the pixel space to obtain the diffusion image.

9. The method according to any one of claims 1 to 8, characterized in that When the sample category is the category to which the sample image as a whole belongs, the sample feature of the sample image under the sample category is the overall image feature of the sample image under the sample category, the training semantics of the sample identifier is used to express the overall image feature of the sample image under the sample category, and the diffusion image corresponding to the sample image is an image generated by diffusion based on the overall sample image; When the sample category is the category to which the sample object in the sample image belongs, the sample features of the sample image under the sample category are the object features of the sample object in the sample image under the sample category, the training semantics of the sample identifier is used to express the object features of the sample object in the sample image under the sample category, and the diffusion image corresponding to the sample image is an image generated by diffusion based on the sample object in the sample image.

10. The method according to any one of claims 1 to 9, characterized in that: In the iterative training process of the image generation model, the noise images corresponding to the sample images used in each training process of the image generation model are different; the noise images corresponding to the sample images are different means that the label noise added by the noise addition process of the sample images in each training process of the image generation model is different; The different labeling noises include any one of the following: the types of labeling noises added in each training process of the image generation model are different; the types of labeling noises added in each training process of the image generation model are the same but the intensities are different.

11. An image processing device, characterized in that: include: An acquisition unit, configured to acquire sample data for training an image generation model, wherein the sample data includes a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image; a processing unit, configured to call the image generation model to extract training representation information of the sample identifier, wherein the training representation information of the sample identifier is used to represent the training semantics of the sample identifier, and the training semantics of the sample identifier is used to express the sample features of the sample image under the sample category; The processing unit is further used to call the image generation model based on the training representation information of the sample identifier to predict the noise added to the noise image corresponding to the sample image to obtain the predicted noise; The noise image is obtained by performing noise processing on the sample image, and the noise added by the noise processing is label noise; The processing unit is further used to update the model parameters of the image generation model according to the difference between the marked noise and the predicted noise, and the difference between the training semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model; The trained image generation model is used to perform denoising on the noisy image based on the text description information containing the sample identifier, and generate a diffusion image semantically related to the text description information.

12. A computer device, characterized in that: The computer device comprises: a processor suitable for implementing a computer program; A computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by the processor and executing the image processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the image processing method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 10 is implemented.

Images