Image processing method and device, computer equipment, storage medium and program product

Abstract

The invention provides an image processing method and device, computer equipment, a storage medium and a program product. The method relates to the technical field of computer vision of artificial intelligence, and comprises the following steps: acquiring sample data which comprises a sample image, a sample category to which the sample image belongs and a sample identifier set for the sample image; calling an image generation model to extract representation information of the sample identifier; calling an image generation model to predict noise added to a noise image corresponding to the sample image based on the representation information of the sample identifier to obtain predicted noise; the noise image is obtained by adding marking noise to the sample image; and according to the difference between the marked noise and the predicted noise and the difference between the semantics of the sample identifier and the semantics of the sample category, updating model parameters of the image generation model so as to train the image generation model. By adopting the embodiment of the invention, the overfitting degree of the image generation model can be reduced, and the training effect of the image generation model is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of artificial intelligence technology, and specifically to an image processing method, an image processing apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of artificial intelligence technology, image generation models (also known as diffusion models) have made significant progress in the field of image generation, and the creative potential of artificial intelligence has been released. The application scenarios of image generation models are usually as follows: specific features (for example, a specific dog in an image) can be embedded into the image generation model to generate diffusion images related to specific features (for example, a specific dog in an image standing on the beach). Ideally, multiple sample images of the same category are required to train the image generation model, but in actual situations, multiple sample images of the same category are often lacking, and a single sample image is generally used to train the image generation model. However, using a single sample image to train the image generation model often results in the problem of overfitting, resulting in poor training results for the image generation model. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, apparatus, computer equipment, storage medium, and program product, which can reduce the overfitting degree of the image generation model and improve the training effect of the image generation model.

[0004] On the one hand, an embodiment of the present application provides an image processing method, the image processing method comprising:

[0005] Acquire sample data for training an image generation model, the sample data including a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image;

[0006] Calling the image generation model to extract representation information of the sample identification, the representation information of the sample identification is used to represent the semantics of the sample identification, and the semantics of the sample identification is used to express the sample characteristics of the sample image under the sample category;

[0007] The image generation model is called based on the representation information of the sample identification to predict the noise added to the noise image corresponding to the sample image to obtain the predicted noise; the noise image is obtained by adding noise to the sample image, and the noise added by the noise adding process is the label noise;

[0008] According to the difference between the labeled noise and the predicted noise, and the difference between the semantics of the sample identifier and the semantics of the sample category, the model parameters of the image generation model are updated to train the image generation model; the trained image generation model is used to denoise the noisy image based on the text description information containing the sample identifier, and generate a diffusion image semantically related to the text description information.

[0009] Accordingly, an embodiment of the present application provides an image processing device, the image processing device comprising:

[0010] An acquisition unit, used to acquire sample data for training an image generation model, the sample data including a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image;

[0011] A processing unit, used to call the image generation model to extract representation information of the sample identification, where the representation information of the sample identification is used to represent the semantics of the sample identification, and the semantics of the sample identification is used to express the sample characteristics of the sample image under the sample category;

[0012] The processing unit is further used to call the image generation model based on the representation information of the sample identification, predict the noise added to the noise image corresponding to the sample image, and obtain the predicted noise; the noise image is obtained by performing noise processing on the sample image, and the noise added by the noise processing is the label noise;

[0013] The processing unit is also used to update the model parameters of the image generation model according to the difference between the marked noise and the predicted noise, and the difference between the semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model; the trained image generation model is used to denoise the noisy image based on the text description information containing the sample identifier, and generate a diffusion image semantically related to the text description information.

[0014] In one implementation, the training steps of extracting the representation information of the sample identification, predicting the noise added to the noisy image, and updating the model parameters of the image generation model are used to train the image generation model once; the processing unit is also used to perform the following steps:

[0015] According to the training steps, the image generation model is iteratively trained until the training termination condition is reached, thereby obtaining a trained image generation model;

[0016] The iterative training process of the image generation model takes the initial representation information of the sample identifier as the training starting point; wherein, the initial representation information of the sample identifier is determined by minimizing the difference between the semantics of the sample image, the semantics of the sample category and the native semantics of the sample identifier. The training starting point means: taking the initial representation information of the sample identifier as the representation information of the sample identifier and participating in the first training process of the image generation model.

[0017] In one implementation, during the iterative training process of the image generation model, the noise images corresponding to the sample images used in each training process of the image generation model are different; the noise images corresponding to the sample images are different in that: the labeling noise added to the sample images by the noise processing in each training process of the image generation model is different;

[0018] The different labeling noises include any one of the following: the types of labeling noises added in each training process of the image generation model are different, and the types of labeling noises added in each training process of the image generation model are the same but the intensities are different.

[0019] In one implementation, the processing unit is used to determine the initial representation information of the sample identifier by minimizing the difference between the semantics of the sample image, the semantics of the sample category and the native semantics of the sample identifier, and specifically performs the following steps:

[0020] Extracting representation information of the sample image, where the representation information of the sample image is used to represent the semantics of the sample image;

[0021] Extracting the representation information of the sample category, the representation information of the sample category is used to represent the semantics of the sample category;

[0022] Averaging the representation information of the sample image and the representation information of the sample category to obtain average representation information;

[0023] Extracting native representation information of the sample identifier, where the native representation information of the sample identifier is used to represent the native semantics of the sample identifier;

[0024] The initial representation information of the sample identifier is determined by minimizing the distance between the native representation information of the sample identifier and the average representation information.

[0025] In one implementation, the processing unit, when used to extract the representation information of the sample image, is specifically used to perform the following steps:

[0026] Extracting original image representation information of the sample image, where the original image representation information is used to represent the original image semantics of the sample image;

[0027] Extracting background-removed representation information of the background-removed image corresponding to the sample image, where the background-removed representation information is used to represent the semantics of the background-removed image;

[0028] The original image representation information and the background removal representation information are determined as the representation information of the sample image.

[0029] In one implementation, the image generation model includes a text encoder and an image diffusion sub-model, the text encoder is used to extract representation information of sample identification, and the image diffusion sub-model is used to predict noise; a training module is added to the text encoder and the image diffusion sub-model;

[0030] The processing unit is used to update the model parameters of the image generation model according to the difference between the marked noise and the predicted noise, and the difference between the semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model, and is specifically used to perform the following steps:

[0031] According to the difference between labeling noise and prediction noise, and the difference between the semantics of sample identification and the semantics of sample category, the original model parameters in the text encoder and image diffusion sub-model are kept unchanged, and the model parameters of the training modules added in the text encoder and image diffusion sub-model are updated to train the image generation model.

[0032] In one implementation, predicting the noise added to the noisy image is performed by calling an image diffusion sub-model in the image generation model, the image diffusion sub-model includes N cross-attention modules, any one of the N cross-attention modules is represented as an i-th cross-attention module, N is an integer greater than or equal to 2, and i is a positive integer less than or equal to N;

[0033] The processing unit is used to call the image generation model based on the representation information of the sample identification, predict the noise added to the noise image corresponding to the sample image, and when the predicted noise is obtained, specifically perform the following steps:

[0034] When i=1, the i-th cross attention module is called to perform correlation calculation on the representation information of the sample identifier and the noise image to obtain the cross attention result of the i-th cross attention module;

[0035] When i>1, the i-th cross attention module is called to perform correlation calculation on the cross attention result of the i-1-th cross attention module and the representation information of the sample identifier to obtain the cross attention result of the i-th cross attention module;

[0036] The cross-attention result of the Nth cross-attention module is determined as the prediction noise.

[0037] In one implementation, the trained image generation model includes a trained text encoder and a trained image diffusion sub-model; the acquisition unit is further used to perform the following steps:

[0038] Get text description information including sample identification;

[0039] The processing unit is further configured to perform the following steps:

[0040] Calling the trained text encoder to extract representation information of the text description information; the representation information of the text description information is used to express the semantics of the text description information;

[0041] The trained image diffusion sub-model is called to denoise the noisy image based on the representation information of the text description information, and a diffusion image corresponding to the sample image is generated. The diffusion image is semantically related to the text description information.

[0042] In one implementation, the noise image is obtained by converting the sample image from the pixel space to the coding space and then performing noise processing in the coding space; the processing unit is used to call the trained image diffusion sub-model, perform denoising processing on the noise image based on the representation information of the text description information, and generate a diffusion image corresponding to the sample image, specifically for performing the following steps:

[0043] In the coding space, calling the trained image generation model based on the representation information of the text description information, denoising the noisy image, and obtaining the coding information of the diffusion image;

[0044] The coding information of the diffusion image is converted from the coding space to the pixel space to obtain the diffusion image.

[0045] In one implementation, when the sample category is the category to which the sample image as a whole belongs, the sample feature of the sample image under the sample category is the overall image feature of the sample image under the sample category, the semantics of the sample identifier is used to express the overall image feature of the sample image under the sample category, and the diffusion image corresponding to the sample image is an image generated by diffusion based on the overall sample image;

[0046] When the sample category is the category to which the sample object in the sample image belongs, the sample features of the sample image under the sample category are the object features of the sample object in the sample image under the sample category, the semantics of the sample identifier is used to express the object features of the sample object in the sample image under the sample category, and the diffusion image corresponding to the sample image is an image generated by diffusion based on the sample object in the sample image.

[0047] Accordingly, an embodiment of the present application provides a computer device, the computer device comprising:

[0048] a processor suitable for implementing a computer program;

[0049] A computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the above-mentioned image processing method.

[0050] Accordingly, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is read and executed by a processor of a computer device, the computer device executes the above-mentioned image processing method.

[0051] Accordingly, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the above-mentioned image processing method.

[0052] In an embodiment of the present application, in the process of training an image generation model using a single sample image, a sample identifier can be set for the sample image, and the semantics of the sample identifier can be used to express the sample features of the sample image under the sample category. Based on the difference between the semantics of the sample identifier and the semantics of the sample category, the model parameters of the image generation model are updated, and the image generation model is trained. This can improve the similarity between the semantics of the sample identifier and the semantics of the sample category. In other words, the similarity between the sample features of the sample image under the sample category and the semantics of the sample category can be improved. This is conducive to reducing the overfitting degree of the image generation model and improving the training effect of the image generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 It is a schematic diagram of the principle of an image generation model provided in an embodiment of the present application;

[0055] Figure 2 is a schematic diagram of the architecture of an image generation system provided in an embodiment of the present application;

[0056] Figure 3 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0057] Figure 4 It is a structural schematic diagram of an image generation model in a training phase provided in an embodiment of the present application;

[0058] Figure 5 It is a structural schematic diagram of an image generation model in which a training module is added in a training phase provided in an embodiment of the present application;

[0059] Figure 6 It is a schematic diagram of a process for determining initial characterization information of a sample identifier provided in an embodiment of the present application;

[0060] Figure 7 is a flowchart of another image processing method provided in an embodiment of the present application;

[0061] Figure 8 is a structural schematic diagram of a trained image generation model provided in an embodiment of the present application;

[0062] Fig. 9 is a schematic diagram of the structure of another trained image generation model provided in an embodiment of the present application;

[0063] Fig.10 is a comparative schematic diagram of an image generation effect provided in an embodiment of the present application;

[0064] Fig.11 is a comparative schematic diagram of another image generation effect provided in an embodiment of the present application;

[0065] Fig.12 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0066] Fig.13 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0068] In order to more clearly understand the technical solution provided by the embodiments of the present application, the key terms involved in the embodiments of the present application are first introduced here:

[0069] The embodiments of the present application relate to an image generation model. The image generation model may also be referred to as a diffusion model, which refers to a model that can generate a diffusion image semantically related to the text description information based on the given sample image, the sample category (also referred to as the concept category) to which the sample image belongs, and the text description information, and diffuses the sample features of the sample image under the sample category. More specifically:

[0070] like Figure 1 As shown, the training principle of the image generation model 101 can be roughly described as follows: a sample image 102 (for example, Figure 1 ), sample category 103 to which the sample image 102 belongs (e.g., Figure 1 The sample category "hamster" in the example, or can be expressed in English as the sample category "hamster") and the sample identifier 104 set for the sample image 102 (for example, Figure 1 The training goal of the image generation model 101 is to improve the similarity between the semantics (also called embedded concepts) of the sample identifier 104 extracted by the image generation model 101 and the sample features of the sample image 102 under the sample category 103; for example, Figure 1 In the example, the training goal of the image generation model 101 is to improve the similarity between the semantics of “S*” extracted by the image generation model 101 and the features of “hamster” in the sample image 102. After the training of the image generation model 101 is completed, the semantics of the sample identifier 104 extracted by the trained image generation model 101 can more accurately express the sample features of the sample image 102 under the sample category 103.

[0071] like Figure 1 As shown, the application of the image generation model 101 can be roughly described as follows: the input of the trained image generation model 101 is text description information 105 including sample identification 104 (for example, Figure 1 The trained image generation model 101 can diffuse the semantics of the sample identifier 104 (i.e., the sample features of the sample image 102 under the sample category 103) based on the sample identifier 104 to generate a diffused image 106 corresponding to the sample image 102. The generated diffused image 106 can include the sample features of the sample image 102 under the sample category 103, and can also be semantically related to other description information in the text description information 105 except the sample identifier 104 (for example, "one", "in the bucket", and other description information).

[0072] It should be noted that the image generation model involved in the embodiment of the present application is a model with basic image generation capabilities. The image generation model with basic image generation capabilities means that: based on given description information, the image generation model has the ability to generate images that are semantically related to the given description information; it can be understood that the image generation model provided in the embodiment of the present application is a pre-trained image generation model, and the training of the image generation model in the embodiment of the present application can be understood as fine-tuning the image generation model, with the help of the basic image generation capability that the image generation model already possesses, based on the sample features of the sample image under the sample category, combined with the text description information for diffusion, to generate a diffuse image corresponding to the sample image.

[0073] It should also be noted that the image generation model involved in the embodiments of the present application may be a machine learning model related to the field of computer vision technology of artificial intelligence.

[0074] Artificial Intelligence (AI) refers to the theories, methods, technologies and application systems that use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence.

[0075] Computer Vision (CV) technology is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further process images so that the computer processes them into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data.

[0076] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching. The pre-trained model is the latest development of deep learning, which integrates the above technologies.

[0077] Furthermore, the embodiments of the present application do not limit the type of image generation model. The image generation model is a model with basic image generation capabilities. The image generation model may include but is not limited to any of the following: Diffusion Probabilistic Model (Diffusion Probabilistic Model, which may be referred to as Diffusion Model), Denoising Diffusion Probabilistic Model, and Stable-Diffusion Model.

[0078] In the training process of the image generation model, there is often a lack of multiple sample images under the same category. A single sample image is generally used to train the image generation model. However, the use of a single sample image to train the image generation model often leads to the problem of overfitting, resulting in poor training effect of the image generation model. Based on this, the embodiment of the present application proposes an image processing method that can reduce the overfitting degree of the image generation model and improve the training effect of the image generation model. Specifically, the image processing method proposed in the embodiment of the present application can train the image generation model based on the difference between the semantics of the sample identifier and the semantics of the sample category, and can improve the similarity between the semantics of the sample identifier and the semantics of the sample category. The semantics of the sample identifier can be used to express the sample features of the sample image under the sample category, that is, the similarity between the sample features of the sample image under the sample category and the semantics of the sample category can be improved, which is conducive to reducing the overfitting degree of the image generation model and improving the training effect of the image generation model. In addition, the image generation model proposed in the embodiment of the present application can also initialize the representation information of the sample identifier. Initializing the representation information of the sample identifier refers to finding the initial representation information of the sample identifier that is close to the semantics of the sample image and the semantics of the sample category to which the sample image belongs, as the training starting point of the image generation model. This can quickly guide the image generation model in the appropriate training direction, improve the fitting efficiency of the image generation model, and improve the generalization of the image generation model.

[0079] An image processing system suitable for implementing the image processing method provided in an embodiment of the present application is introduced below in conjunction with the accompanying drawings.

[0080] like Figure 2 As shown, the image processing system may include a model training device 201 and a model application device 202, and a direct communication connection may be established between the model training device 201 and the model application device 202 by wired communication, or an indirect communication connection may be established between the model training device 201 and the model application device 202 by wireless communication.

[0081] The model training device 201 can be used to train an image generation model. Specifically, a pre-trained image generation model is deployed in the model training device 201. The model training device 201 can train the image generation model (i.e., fine-tune the image generation model) based on sample data (the sample data may include a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image), so that the semantics of the sample identifier extracted by the trained image generation model can more accurately express the sample features of the sample image under the sample category, thereby enabling the trained image generation model to have the ability to perform image diffusion based on the sample features of the sample image under the sample category.

[0082] The model application device 202 can be used to call the trained image generation model for image diffusion. Specifically, the trained image generation model can be deployed in the model application device 202, and the model application device 202 can generate a diffusion image corresponding to the sample image based on the given text description information (the text description information includes the sample identifier) ​​and the semantics of the sample identifier (i.e., the sample features of the sample image under the sample category) as the diffusion basis.

[0083] The model training device 201 can be a terminal or a server, and the model application device 202 can be a terminal or a server. The terminal mentioned in the embodiment of the present application may include any of the following: a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device, a smart watch, a vehicle terminal, a smart home appliance, and an aircraft, etc., but is not limited thereto. The server mentioned in the embodiment of the present application may be a separate physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, which are not limited in the embodiment of the present application.

[0084] It should be noted that Figure 2 The image processing system shown is described by taking the model training device 201 and the model application device 202 as an example. In an actual application scenario, the model training device 201 and the model application device 202 in the image processing system can be as follows: Figure 2 As shown, the model training device 201 and the model application device 202 in the image processing system may be integrated into different devices, or the model training device 201 and the model application device 202 in the image processing system may be integrated into the same device, for example, the model training device 201 and the model application device 202 may be integrated into the same terminal, or the same server. When the model training device 201 and the model application device 202 in the image processing system are integrated into the same device, the integrated device of the model training device 201 and the model application device 202 may be used to train the image generation model, and may also be used to call the trained image generation model for image diffusion.

[0085] Understandably, Figure 2 The image processing system shown is intended to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems.

[0086] The image processing method provided in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0087] The present application provides an image processing method, which mainly introduces the training process of the image generation model (including: the structure of the image generation model, the representation information of the initialized sample identification, and the loss information formed by the difference between the semantics of the sample identification and the semantics of the sample category, etc.). The image processing method can be executed by a computer device, which can be, for example, the above-mentioned Figure 2 The model training device 201 in the image processing system shown in FIG. 1 may be an integrated device of the model training device 201 and the model application device 202. Figure 3 As shown, the image processing method may include but is not limited to the following steps S301 to S304:

[0088] S301, obtaining sample data for training an image generation model, where the sample data includes a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image.

[0089] The sample data used to train the image generation model may include: a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image. The sample image is a single image used to train the image generation model. The sample category to which the sample image belongs may be set according to the task type of the image generation task; if the task type of the image generation task is to perform image diffusion based on the sample image as a whole, then the sample category to which the sample image belongs may be the category to which the sample image as a whole belongs, for example, "landscape painting", "oil painting", etc.; if the task type of the image generation task is to perform image diffusion based on the sample object in the sample image, then the sample category described in the sample image may be the sample category to which the sample object in the sample image belongs, for example, "dog", "hamster", etc. The purpose of setting the sample identifier for the sample image is to expect that the semantics of the sample identifier extracted by the trained image generation model can more accurately express the sample features of the sample image under the sample category, so that in the application stage of the trained image generation model, based on the text description information containing the sample identifier, a better diffusion effect of the sample image can be achieved.

[0090] Steps S302 to S304 in the present embodiment introduce the training process of the image generation model. Figure 4 The structure of the image generation model in the model training stage is introduced so that when introducing the training process of the image generation model in steps S302 to S304, reference can be made to Figure 4 The structure of the image generation model shown in Figure 4As shown, the image generation model may include at least an image compression encoder 401, a first text encoder 402, a second text encoder 403, and an image diffusion sub-model 404. Among them:

[0091] (1) Image compression encoder 401: The image compression encoder can be used to convert an image (e.g., a sample image) from a pixel space to a coding space (the coding space can also be called a latent space) to obtain coding information of the sample image. Furthermore, the sample image can be subjected to noise processing in the coding space. Specifically, after the image compression encoder converts the sample image from a pixel space to a coding space to obtain coding information of the sample image, the coding information of the sample image can be subjected to noise processing to obtain a noisy image corresponding to the sample image.

[0092] The embodiment of the present application does not limit the type of image compression encoder. The embodiment of the present application is described by taking the image compression encoder being a compression encoder (Encoder) in a VAE (Variational Autoencoder) as an example.

[0093] (2) First text encoder (first text encoder) 402: The first text encoder can be used to extract representation information of the sample identifier, and the representation information of the sample identifier can be used to represent the semantics of the sample identifier. The embodiment of the present application does not limit the type of the first text encoder. The embodiment of the present application takes the first text encoder as a text encoder in the CLIP model as an example for explanation.

[0094] (3) Second text encoder (second text encoder) 403: The second text encoder can be used to extract representation information of the sample category, and the representation information of the sample category can be used to represent the semantics of the sample category. The embodiment of the present application does not limit the type of the second text encoder. The embodiment of the present application takes the second text encoder as a text encoder in the CLIP model as an example for explanation.

[0095] (4) Image diffusion sub-model (UNet) 404: During the training phase of the image generation model, the image diffusion sub-model can be used to predict the noise added to the noise image corresponding to the sample image based on the representation information of the sample identification, and obtain the encoding information of the predicted noise.

[0096] based on Figure 4The illustrated embodiment provides an introduction to the structure of the image generation model. The general training principle of the image generation model is as follows: ① calling the image compression encoder 401 to convert the sample image from the pixel space to the coding space to obtain the coding information of the sample image. The coding information of the sample image can be denoised in the coding space to obtain a noisy image corresponding to the sample image. The noise added by the denoising process is the labelling noise; ② calling the first text encoder 402 to extract the representation information of the sample identifier, which can be used to represent the semantics of the sample identifier; calling the second text encoder 403 to extract the representation information of the sample category, which can be used to identify the semantics of the sample category; ③ calling the image diffusion sub-model 404 to predict the noise added to the noise image corresponding to the sample image based on the representation information of the sample identifier to obtain the predicted noise; ④ based on the difference between the predicted noise and the labeled noise, and the difference between the representation information of the sample identifier and the representation information of the sample category (i.e., the difference between the semantics of the sample identifier and the semantics of the sample category), updating the parameters of the image generation model to train the image generation model. The above ①-④ describe any training process of the image generation model. The image generation model needs to be iteratively trained according to the above ①-④ until the training termination condition is met to obtain a trained image generation model. Among them, meeting the training termination condition may include any of the following: the number of training times of iterative training reaches the number threshold (the number threshold can be set according to an empirical value to control the number of training times, for example, the number threshold is 100 times), and the loss information of the image generation model is less than the loss threshold (the loss threshold can be set according to an empirical value to control the loss information of the image generation model).

[0097] In combination Figure 4 After introducing the structure of the image generation model and the general training principle of the image generation model, the training process of the image generation model is introduced in detail in combination with steps S302 to S304.

[0098] S302: Calling an image generation model to extract representation information of a sample identifier.

[0099] In step S302, the image generation model is called to extract representation information of the sample identifier, specifically, the text encoder in the image generation model is called (the text encoder here specifically refers to the first text encoder) to extract the representation information of the sample identifier. The representation information of the sample identifier can be used to represent the semantics of the sample identifier, and the semantics of the sample identifier can be used to express the sample characteristics of the sample image under the sample category.

[0100] Specifically, if Figure 4As shown, the text encoder may include a tokenizer layer, an embedding layer, and a text transformer layer. The tokenizer layer may be called to perform word segmentation on the sample identifier to obtain one or more word segments contained in the sample identifier. Then, the tokenizer layer may be called to map each word segment to a token corresponding to each word segment. The token is a number that can be recognized by a computer device. Then, the embedding layer may be called to convert the token of each word segment into a corresponding embedding vector. Then, the transformer layer may be called to perform semantic understanding on the embedding vectors of each word segment to obtain representation information of the sample identifier.

[0101] S303, calling the image generation model based on the representation information of the sample identifier, predicts the noise added to the noise image corresponding to the sample image, and obtains the predicted noise.

[0102] In step S303, the image generation model is called to predict the noise added to the noise image corresponding to the sample image based on the representation information of the sample identifier to obtain the predicted noise. Specifically, the image diffusion submodel in the image generation model is called to predict the noise added to the noise image corresponding to the sample image based on the representation information of the sample identifier to obtain the predicted noise. As described above, the noise image corresponding to the sample image is obtained by performing noise processing on the coding information of the sample image in the coding space after converting the sample image from the pixel space to the coding space. The process of predicting noise also occurs in the coding space. That is to say, the image generation model (specifically the image diffusion submodel in the image generation model) can be called in the coding space to predict the noise added to the noise image based on the representation information of the sample identifier to obtain the predicted noise. During the training process of the image processing model in the embodiment of the present application, the sample image is converted from the pixel space to the coding space, the sample image is denoised in the coding space, and the noise is predicted for the noisy image corresponding to the sample image in the coding space. This is because the dimension of the pixel space is higher than that of the coding space. Noising the sample image in the coding space and predicting the noise for the noisy image corresponding to the sample image in the coding space can reduce the amount of data processed during the denoising process and the noise prediction process, and can improve the efficiency of the denoising process and the noise prediction process, thereby improving the training efficiency of the image generation model to a certain extent.

[0103] Furthermore, the image diffusion submodel may include N cross-attention modules, any one of the N cross-attention modules may be represented as the i-th cross-attention module, N is an integer greater than or equal to 2, and i is a positive integer less than or equal to N; in the encoding space, the image generation model (specifically the image diffusion submodel in the image generation model) is called to predict the noise added to the noise image based on the representation information of the sample identifier to obtain the predicted noise, which may specifically include: when i=1, calling the i-th cross-attention module to perform correlation calculation on the representation information of the sample identifier and the noise image to obtain the cross-attention result of the i-th cross-attention module; when i>1, calling the i-th cross-attention module to perform correlation calculation on the cross-attention result of the i-1-th cross-attention module and the representation information of the sample identifier to obtain the cross-attention result of the i-th cross-attention module; the cross-attention result of the N-th cross-attention module is determined as the predicted noise.

[0104] S304: updating the model parameters of the image generation model according to the difference between the labeled noise and the predicted noise, and the difference between the semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model.

[0105] After the image generation model is called based on the representation information of the sample identifier to predict the noise added to the noise image corresponding to the sample image and obtain the predicted noise, the model parameters of the image generation model can be updated according to the difference between the labeled noise and the predicted noise, and the difference between the semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model. Specifically, the process of updating the model parameters of the image generation model according to the difference between the labeled noise and the predicted noise, and the difference between the semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model, can specifically include:

[0106] (1) The first loss information may be determined according to the difference between the marked noise and the predicted noise. The first loss information may be specifically referred to in the following formula 1:

[0107] L LDM =E ε(x) ,y,∈~N(0,1),t[||∈-∈ θ (z t ,t,c)|| 2 ] Formula 1

[0108] In the above formula 1: L LDM represents the first loss information; x represents the sample image; c represents the representation information of the sample identification; t represents the time step; ∈ represents the label noise; ∈ θ represents the prediction noise; zt It represents the noise image corresponding to the sample image obtained by adding noise to the coding information of the sample image in the coding space at time step t.

[0109] (2) The second loss information may be determined according to the difference between the representation information of the sample identifier and the representation information of the sample category (i.e., the difference between the semantics of the sample identifier and the semantics of the sample category). The second loss information may be specifically referred to in the following formula 2:

[0110]

[0111] In the above formula 2: L CL represents the second loss information; T represents a text encoder (including the first text encoder or the second text encoder); c p represents the sample identification, T(c p ) represents the representation information of the sample identification extracted by calling the text encoder (the text encoder here specifically refers to the first text encoder); c c represents the sample category, T(c c ) represents the representation information of the sample category extracted by calling the text encoder (the text encoder here specifically refers to the second text encoder); α cl Represents a weight parameter. The embodiment of the present application does not limit the specific parameter value of the weight parameter. The specific parameter value of the weight parameter can be set according to an empirical value. For example, the parameter value of the weight parameter is set to 1.

[0112] The representation information of the sample category can be extracted by calling the text encoder in the image generation model (the text encoder here specifically refers to the second text encoder). The process of the second text encoder extracting the representation information of the sample category is similar to the process of the first text encoder extracting the representation information of the sample identifier in the above step S302. For details, please refer to the process of the first text encoder extracting the representation information of the sample identifier in the above step S302, which will not be repeated here. The representation information of the sample category can be used to represent the semantics of the sample category.

[0113] (3) The first loss information and the second loss information may be weighted and summed to obtain the loss information of the image generation model. The embodiment of the present application does not limit the specific weight values ​​of the first weight used to weight the first loss information and the second weight used to weight the second loss information. The specific weight values ​​of the first weight and the second weight may be determined based on empirical values. For example, the weight values ​​of the first weight and the second weight may be set to 1. For another example, the weight value of the first weight may be set to 1, and the weight value of the second weight may be set to 0.5, etc. Then, the model parameters of the image generation model may be updated in the direction of minimizing the loss information of the image generation model to train the image generation model.

[0114] Furthermore, when updating the model parameters of the image generation model in the direction of minimizing the loss information of the image generation model, there may be two ways of updating the model parameters of the image generation model: the first updating method is to directly update the model parameters of the image generation model, that is, the original model parameters of the image generation model may be updated in the direction of minimizing the loss information of the image generation model to train the image generation model; the second updating method is to add a training module to the image generation model, keep the original model parameters of the image generation model unchanged, and update the model parameters of the training module, that is, the original model parameters of the image generation model may be kept unchanged in the direction of minimizing the loss information of the image generation model, and update the model parameters of the training module to train the image generation model. The first updating method and the second updating method are described in detail below:

[0115] In the first updating method, the model parameters of the first text encoder and the image diffusion sub-model in the image generation model can be updated in the direction of minimizing the loss information of the image generation model, and the model parameters of the image compression encoder and the second text encoder are kept unchanged to train the image generation model. In other words, the essence of the image generation model training of the present application is to train the first text encoder and the image diffusion sub-model in the image generation model.

[0116] In the second update method, if Figure 5 As shown, a training module (LoRAmodule) can be added to the image generation model, and the training module can be added to the text encoder (the text encoder here specifically refers to the first text encoder) for extracting the representation information of the sample identification, and added to the image diffusion submodule; for the text encoder, the training module can be specifically added to the conversion layer of the text encoder, and the training module can be used to reduce the dimension of the representation information of the sample identification output by the conversion layer, and then restore it to the original dimension of the representation information of the sample identification through dimensionality increase processing; for the image diffusion submodel, the image diffusion submodel can include multiple cross attention modules, and the training module can be added to the linear layer of each cross attention module. The training module can be used to reduce the dimension of the result output by the linear layer, and then restore it to the original dimension of the result through dimensionality increase processing. Based on this, the original model parameters in the text encoder and the image diffusion submodel can be kept unchanged in the direction of minimizing the loss information of the image generation model, and the model parameters of the training modules added in the text encoder and the image diffusion submodel can be updated to train the image generation model. That is to say, the essence of the image generation model training in the present application is to train the first text encoder of the image generation model and the training modules added in the image diffusion sub-model.

[0117] In step S304, based on the difference between the semantics of the sample identifier and the semantics of the sample category, the model parameters of the image generation model are updated to train the image generation model, which can improve the similarity between the semantics of the sample identifier and the semantics of the sample category, associate the semantics of the sample identifier with the semantics of the sample category, and better utilize the prior knowledge of the image generation model (i.e., the semantics of the sample category), so as to reduce the overfitting degree of the image generation model and improve the training effect of the image generation model. In the first updating method, directly updating the original model parameters of the image generation model can reduce the overfitting degree of the image generation model and improve the training effect of the image generation model. However, under normal circumstances, the model parameters of the image generation model are relatively large, and directly updating the original model parameters of the image generation model may lose a certain model training efficiency. In the second updating method, compared with the original model parameters of the image generation model, the model parameters of the training module are small, and updating the model parameters of the training module can reduce the overfitting degree of the image generation model, improve the training effect of the image generation model, and can also improve a certain model training efficiency.

[0118] It should be noted that in the above steps S301 to S304, the training steps of extracting the representation information of the sample identifier, predicting the noise added to the noisy image, and updating the model parameters of the image generation model can be used to train the image generation model once. The image generation model can be iteratively trained according to the above training steps until the training termination condition is reached to obtain a trained image generation model. The iterative training process of the image generation model takes the initial representation information of the sample identifier as the training starting point. The initial representation information of the sample identifier can be determined by minimizing the difference between the semantics of the sample image, the semantics of the sample category, and the native semantics of the sample identifier. The training starting point can specifically refer to: using the initial representation information of the sample identifier as the representation information of the sample identifier to participate in the first training process of the image generation model. It can be seen that by minimizing the differences between the semantics of the sample image, the semantics of the sample category and the native semantics of the sample identifier, the semantics of the sample identifier represented by the initial representation information of the determined sample identifier can be made close to the semantics of the sample image and the semantics of the sample category. The initial representation information of the sample identifier is used to participate in the first training of the image generation model, which can quickly guide the image generation model in the appropriate training direction, improve the fitting efficiency of the image generation model, and improve the generalization of the image generation model.

[0119] Furthermore, the semantics of the sample image can be represented by the representation information of the sample image, and the representation information of the sample image can be obtained by semantically understanding the sample image using an image semantic encoder. The embodiment of the present application does not limit the type of the image semantic encoder. The embodiment of the present application assumes that the image semantic encoder is the Image Semantic Encoder in the CLIP (Contrastive Language-Image Pre-Training) model. Encoder (image encoder) is used as an example for explanation. The image semantic encoder can be a trained image encoder that already has semantic understanding capabilities. The semantics of the sample category can be represented by the representation information of the sample category, and the representation information of the sample category can be extracted by the second text encoder using the image generation model. The native semantics of the sample identifier can be represented by the native representation information of the sample identifier, and the native representation information of the sample identifier can be extracted for the first time by the first text encoder using the image generation model. Based on this, the process of determining the initial representation information of the sample identifier by minimizing the difference between the semantics of the sample image, the semantics of the sample category and the native semantics of the sample identifier can specifically include: extracting the representation information of the sample image, and the representation information of the sample image can be used to represent the semantics of the sample image; extracting the representation information of the sample category, and the representation information of the sample category can be used to represent the semantics of the sample category; averaging the representation information of the sample image and the representation information of the sample category to obtain average representation information; extracting the native representation information of the sample identifier, and determining the initial representation information of the sample identifier by minimizing the distance between the native representation information of the sample identifier and the average representation information.

[0120] Optionally, the representation information of the sample image may include original image representation information of the sample image and background-removed representation information of a background-removed image corresponding to the sample image. Specifically, the original image representation information of the sample image may be extracted, and the original image representation information of the sample image may be used to represent the original image semantics of the sample image, and the original image representation information of the sample image may be obtained by performing semantic understanding on the sample image using an image semantic encoder; the background-removed representation information of the background-removed image corresponding to the sample image may be extracted, and the background-removed representation information may be used to represent the semantics of the background-removed image, and the background-removed image may be obtained after removing the background from the sample image, and the background-removed representation information may be obtained by performing semantic understanding on the background-removed image using an image semantic encoder; the original image representation information and the background-removed representation information may be determined as the representation information of the sample image. Further, as Figure 6 As shown, the sample image (for example, Figure 6 The original image representation information of the sample image (for example, the image containing the hamster and the background) and the background-removed image corresponding to the sample image (for example, Figure 6The background-removed representation information and sample categories (e.g., images containing hamsters but not background) are Figure 6 The representation information of the sample category "hamster" in the sample is averaged to obtain the average representation information. Then, the sample identifier (for example, Figure 6 The original representation information of the sample identifier "S*" in the sample identifier is determined by minimizing the distance between the original representation information of the sample identifier and the average representation information. For details, see the following formula 3:

[0121]

[0122] In the above formula 3, L PE represents the initial representation information of the sample identification; T represents a text encoder (including the first text encoder or the second text encoder); c p represents the sample identification, T(c p ) represents the original representation information of the sample identifier extracted by calling the text encoder (the text encoder here specifically refers to the first text encoder); c c represents the sample category, T(c c ) represents the representation information of the sample category extracted by calling the text encoder (the text encoder here specifically refers to the second text encoder); x represents the sample image, x m represents the background-removed image corresponding to the sample image; I represents the image semantic encoder, I(x) represents the representation information of the sample image obtained by calling the image semantic encoder to perform semantic understanding on the sample image, and I(x) m ) indicates calling the image semantic encoder to perform semantic understanding on the background-removed image corresponding to the sample image, and obtaining the background-removed representation information of the background-removed image; θ m Indicates the rule or algorithm used for averaging; θ m (I(x),I(x m ),T(c c )) represents the average representation information.

[0123] It is easy to see that in the method of not introducing the background-removed image corresponding to the sample image when determining the initial representation information of the sample identification, more attention is paid to the overall sample image, and this method is more suitable for image generation tasks based on the overall sample image for image diffusion; in the method of introducing the background-removed image corresponding to the sample image when determining the initial representation information of the sample identification, not only the overall sample image but also the sample object in the sample image is paid attention to (usually, the background-removed image obtained after background removal processing of the sample image contains the sample object in the sample image), and this method is more suitable for image generation tasks based on the sample object in the sample image for image diffusion. In other words, the initial representation information of the sample identification can be determined according to the determination method corresponding to the task type of the image generation task, so that the image generation model trained based on the initial representation information of the sample identification can be more adapted to the image generation task.

[0124] It is worth noting that, during the iterative training process of the image generation model, the noise images corresponding to the sample images used in each training process of the image generation model may be different; the different noise images corresponding to the sample images may specifically refer to: the label noise added to the sample images for noise processing in each training process of the image generation model is different. Among them, the different label noises may include any of the following: the types of label noises added in each training process of the image generation model are different (for example, the label noise added in the first training process is Gaussian noise, and the label noise added in the second training process is gamma noise...), the types of label noises added in each training process of the image generation model are the same but the intensities are different (for example, the label noises added in each training process are Gaussian noise, and the intensity of Gaussian noise gradually increases with the increase in the number of training times). By using different noise images in each training process of the image generation model (i.e., using different label noises for noise processing), the image generation model can be trained to achieve better image generation effects for different noise images, which is conducive to improving the training effect of the image generation model.

[0125] In an embodiment of the present application, during the training process of the image generation model, by minimizing the difference between the semantics of the sample image, the semantics of the sample category and the native semantics of the sample identifier, the initial representation information of the sample identifier is determined for the first training process of the image generation model, so that the semantics of the sample identifier represented by the determined initial representation information of the sample identifier can be close to the semantics of the sample image and the semantics of the sample category. The initial representation information of the sample identifier is used to participate in the first training of the image generation model, which can quickly guide the image generation model in the appropriate training direction, improve the fitting efficiency of the image generation model, and improve the generalization of the image generation model. Based on the difference between the semantics of the sample identifier and the semantics of the sample category, the model parameters of the image generation model are updated to train the image generation model, which can improve the similarity between the semantics of the sample identifier and the semantics of the sample category, and can associate the semantics of the sample identifier with the semantics of the sample category, so as to better utilize the prior knowledge of the image generation model (i.e., the semantics of the sample category), so as to reduce the overfitting degree of the image generation model and improve the training effect of the image generation model.

[0126] The embodiment of the present application provides an image processing method, which mainly introduces the application process of a trained image generation model (including: an image diffusion process based on text description information, and specific application scenarios of the trained image generation model, etc.). The image processing method can be executed by a computer device, which can be, for example, an integrated device of a model training device 201 and a model application device 202, or the image processing method can be executed collaboratively by the model training device 201 and the model application device 202. In the embodiment of the present application, the devices that collaboratively execute the image processing method are collectively referred to as computer devices. Figure 7 As shown, the image processing method may include but is not limited to the following steps S701 to S707:

[0127] S701, obtaining sample data for training an image generation model, where the sample data includes a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image.

[0128] In the embodiment of the present application, the above-mentioned execution process of step S701 Figure 3 The execution process of step S301 in the embodiment shown is the same. For the specific execution process, please refer to the above Figure 3 The relevant description of step S301 in the illustrated embodiment will not be repeated here.

[0129] S702, calling an image generation model to extract representation information of a sample identifier.

[0130] In the embodiment of the present application, the above-mentioned execution process of step S702 Figure 3The execution process of step S302 in the embodiment shown is the same. For the specific execution process, please refer to the above Figure 3 The relevant description of step S302 in the illustrated embodiment will not be repeated here.

[0131] S703, calling the image generation model based on the representation information of the sample identifier, predicts the noise added to the noise image corresponding to the sample image, and obtains the predicted noise.

[0132] In the embodiment of the present application, the above-mentioned execution process of step S703 Figure 3 The execution process of step S303 in the embodiment shown is the same. For the specific execution process, please refer to the above Figure 3 The relevant description of step S303 in the illustrated embodiment will not be repeated here.

[0133] S704, updating the model parameters of the image generation model according to the difference between the labeled noise and the predicted noise, and the difference between the semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model, wherein the trained image generation model includes a trained text encoder and a trained image diffusion sub-model.

[0134] In the embodiment of the present application, the above-mentioned execution process of step S704 Figure 3 The execution process of step S304 in the embodiment shown is the same. For the specific execution process, please refer to the above Figure 3 The relevant description of step S304 in the illustrated embodiment will not be repeated here.

[0135] In the embodiment of the present application, steps S705 to S707 introduce the application process of the trained image generation model. Figure 8 The structure of the trained image generation model is introduced so that when introducing steps S705-step S707, the structure of the trained image generation model can be used as the basis. When introducing the application process of the trained image generation model in detail, steps S705-step S707 can be referred to. Figure 8 The structure of the trained image processing model is shown in Figure 2. The structure of the trained image generation model is similar to Figure 4 The structure of the image generation model in the model training phase shown is similar, such as Figure 8 As shown, the trained image generation model may at least include an image compression encoder 401, a trained text encoder 801 (the trained text encoder mentioned here specifically refers to a trained first text encoder), a trained image diffusion sub-model 802, and an image compression decoder 803. Among them:

[0136] (1) Image compression encoder 401: The image compression encoder can be used to convert the sample image from the pixel space to the coding space to obtain the coding information of the sample image. In addition, the sample image can be subjected to noise processing in the coding space. Specifically, after the image semantic encoder converts the sample image from the pixel space to the coding space to obtain the coding information of the sample image, the coding information of the sample image can be subjected to noise processing to obtain a noisy image corresponding to the sample image.

[0137] (2) Trained first text encoder 801: The trained first text encoder can be used to extract representation information of text description information. The text description information may include a sample identifier. The representation information of the text description information can be used to represent the semantics of the text description information.

[0138] (3) Trained image diffusion sub-model 802: In the application stage of the trained image generation model, the trained image diffusion sub-model can be used to denoise the noise image corresponding to the sample image based on the representation information of the text description information to obtain the encoding information of the diffusion image corresponding to the sample image.

[0139] (4) Image compression decoder 803: The decoding process of the image compression decoder is the inverse process of the encoding process of the image compression encoder. In the application stage of the trained image generation model, the image compression decoder can be used to convert the encoding information of the diffusion image from the encoding space to the pixel space to obtain the diffusion image.

[0140] The embodiment of the present application does not limit the type of image compression decoder. The embodiment of the present application is described by taking the image compression decoder as a compression decoder (Decoder) in a VAE (Variational Autoencoder) as an example.

[0141] Corresponding to the training stage of the image generation model, if the image generation model is not added with a training module during the training process, that is, when the model parameters are updated, the original model parameters of the image generation model are directly updated, the structure of the trained image generation model is as follows: Figure 8 As shown in Figure 2; if a training module is added to the image generation model during the training process, that is, when the model parameters are updated, the original model parameters of the image generation model are kept unchanged, and when the model parameters of the added training module are updated, the structure of the trained image generation model is as follows: Fig. 9 As shown; Figure 8 and Fig. 9 The difference is that Fig. 9In the structure of the trained image generation model shown, trained training modules are added to the trained text encoder (here the trained text encoder specifically refers to the trained first text encoder) and the trained image diffusion sub-model; corresponding to the training stage of the image generation model, the trained training module can be added to the conversion layer of the trained text encoder, and the trained training module can be used to reduce the dimension of the representation information of the text description information output by the conversion layer, and then restore it to the original dimension of the representation information of the text description information through dimensionality increase processing; the trained image diffusion sub-model may include multiple cross-attention modules, and the trained training module can be added to the linear layer of each cross-attention module, and the trained training module can be used to reduce the dimension of the result output by the linear layer, and then restore it to the original dimension of the result through dimensionality increase processing.

[0142] Based on the above introduction to the structure of the trained image generation model, the general application process of the trained image generation model can be: ① Call the image compression encoder 401 to convert the sample image from the pixel space to the coding space to obtain the coding information of the sample image. The sample image can be denoised in the coding space to obtain the noisy image corresponding to the sample image; ② Call the trained first text encoder 801 to extract the representation information of the text description information. The text description information contains the sample identifier, and the representation information of the text description information can be used to represent the semantics of the text description information; ③ Call the trained image diffusion sub-model 802 to denoise the noise image corresponding to the sample image based on the representation information of the text description information to obtain the coding information of the diffusion image corresponding to the sample image; ④ Call the image compression decoder 803 to convert the coding information of the diffusion image from the coding space to the pixel space to obtain the diffusion image. The diffusion image is semantically related to the text description information.

[0143] In combination Figure 8 and Fig. 9 After introducing the structure of the trained image generation model and the application process of the trained image generation model, the application process of the trained image generation model is described in detail in combination with steps S705 to S707.

[0144] S705: Obtain text description information including the sample identifier.

[0145] S706, calling the trained text encoder to extract representation information of the text description information.

[0146] In step S706, the process of calling the trained text encoder to extract the representation information of the text description information is similar to the above Figure 3The process of calling the first text encoder to extract the representation information of the sample identification in the illustrated embodiment is similar. Specifically, the trained text encoder may include a token layer, an embedding layer, and a conversion layer. The token layer may be called to perform word segmentation processing on the text description information to obtain one or more word segments contained in the text description information, and then the token layer may be called to map each word segment to a token corresponding to each word segment, where a token is a number that can be recognized by a computer device. Then, the embedding layer may be called to convert the token of each word segment into a corresponding embedding vector. Then, the conversion layer may be called to perform semantic understanding on the embedding vectors of each word segment to obtain the representation information of the text description information.

[0147] S707, calling the trained image diffusion sub-model, performing denoising on the noisy image based on the representation information of the text description information, and generating a diffusion image corresponding to the sample image.

[0148] As described above, the noise image corresponding to the sample image is obtained by performing noise processing on the coding information of the sample image in the coding space after converting the sample image from the pixel space to the coding space. The denoising process also occurs in the coding space, that is, in the coding space, the trained image diffusion sub-model can be called to perform denoising on the noise image based on the representation information of the text description information to obtain the coding information of the diffusion image corresponding to the sample image. Then, the image compression decoder can be called to convert the coding information of the diffusion image from the coding space to the pixel space to obtain the diffusion image, and the diffusion image is semantically related to the text description information. In the application process of the trained image processing model of the embodiment of the present application, the sample image is converted from the pixel space to the coding space, the sample image is denoised in the coding space, and the denoising process is performed on the denoised image corresponding to the sample image in the coding space. This is because the dimension of the pixel space is higher than that of the coding space. The sample image is denoised in the coding space, and the denoising process is performed on the denoised image corresponding to the sample image in the coding space, which can reduce the amount of data processed during the denoising and denoising processes, and can improve the efficiency of the denoising and denoising processes, thereby improving the image generation efficiency to a certain extent.

[0149] In step S707, in the coding space, the trained image diffusion sub-model is called to perform denoising on the noisy image based on the representation information of the text description information to obtain the coding information of the diffusion image corresponding to the sample image, which is the same as the above process. Figure 3The process of predicting noise in the illustrated embodiment is similar. Specifically, the trained image diffusion sub-model may include N cross-attention modules, any of the N cross-attention modules may be represented as the i-th cross-attention module, N is an integer greater than or equal to 2, and i is a positive integer less than or equal to N; when i=1, the i-th cross-attention module is called to perform correlation calculation on the representation information of the text description information and the noise image, and obtain the cross-attention result of the i-th cross-attention module; when i>1, the i-th cross-attention module is called to perform correlation calculation on the cross-attention result of the i-1 cross-attention module and the representation information of the text description information, and obtain the cross-attention result of the i-th cross-attention module; the cross-attention result of the N-th cross-attention module is determined as the encoding information of the diffusion image.

[0150] The above steps S705 to S707 introduce in detail the application process of the trained image generation model. On this basis, two types of application scenarios of the trained image generation model are introduced. The two types of application scenarios correspond to the two types of image generation tasks. Specifically:

[0151] If the task type of the image generation task is to diffuse the image based on the sample image as a whole, then the sample category to which the sample image belongs can be the category to which the sample image as a whole belongs; when the sample category is the category to which the sample image as a whole belongs, the sample features of the sample image under the sample category can be the overall image features of the sample image under the sample category, the semantics of the sample identifier can be used to express the overall image features of the sample image under the sample category, and the diffused image corresponding to the sample image can be an image diffused and generated based on the sample image as a whole. For example, in an image enhancement scenario, the goal of the image enhancement scenario is to diffuse a high-quality (e.g., higher definition, fuller image, etc.) sample image based on the sample image. In this case, the sample category can be the category to which the sample image as a whole belongs (e.g., the sample category is "landscape image"), and the text description information containing the sample identifier can be, for example, "a high-quality S*"; based on this, a high-quality sample image can be diffused based on the sample image, which is conducive to rapid image enhancement. For another example, in an image restoration scenario, the goal of the image restoration scenario is to diffuse restored sample images based on sample images. In this case, the sample category can be the category to which the sample image as a whole belongs (for example, the sample category is "landscape image"), and the text description information containing the sample identifier can be, for example, "a restored S*". Based on this, the restored sample image can be diffused based on the sample image, which is conducive to the rapid realization of image restoration. For another example, in an image art creation scenario, the goal of image art creation is to diffuse images that are similar in style to the sample image but have innovative features based on the sample image. In this case, the sample category can be the category to which the sample image as a whole belongs (for example, the sample category is "oil painting"), and the text description information containing the sample identifier can be, for example, "S* of mixed ink painting". Based on this, images that are similar in style to the sample image but have innovative features can be diffused based on the sample image, which helps artists explore new creative directions and forms of expression.

[0152] If the task type of the image generation task is to diffuse the image based on the sample object in the sample image, then the sample category to which the sample image belongs may be the category to which the sample object in the sample image belongs; when the sample category is the category to which the sample object in the sample image belongs, the sample feature of the sample image under the sample category may be the object feature of the sample object in the sample image under the sample category, the semantics of the sample identifier may be used to express the object feature of the sample object in the sample image under the sample category, and the diffused image corresponding to the sample image may be an image diffused and generated based on the sample object in the sample image. For example, in a character design scenario, the goal of character design is to diffuse a virtual character corresponding to the character object based on the character object in the sample image. In this case, the sample category may be the category to which the sample object (i.e., the character object) in the sample image belongs (for example, the sample category is "person"), and the text description information containing the sample identifier may be, for example, "a cartoon S*", "a running cartoon S*", and "a crying S*", etc. Based on this, a virtual character similar to the character object but with different features, actions, expressions, and styles may be diffused based on the character object in the sample image, which has potential applications in role-playing or animation creation. For another example, in a virtual fitting room scenario, the goal of the virtual fitting room is to diffuse person objects wearing different clothes based on the person objects in the sample image. In this case, the sample category may be the category to which the sample object (i.e., the person object) in the sample image belongs (for example, the sample category is "person"), and the text description information containing the sample identifier may be, for example, "S* wearing a denim skirt", "S* wearing a suit", and "S* wearing Martin boots", etc. Based on this, person objects wearing different clothes can be diffused based on the person objects in the sample image, which can help preview the clothing effect and provide personalized shopping suggestions. For example, in a personalized product customization scenario, the goal of personalized products is to diffuse personalized products containing the target object based on the target object in the sample image. In this case, the sample category can be the category to which the animal object (e.g., Corgi) in the sample image belongs (e.g., the sample category is "dog"), and the text description information containing the sample identifier can be, for example, "a cup containing S*", "a mobile phone case containing S*", and "a T-shirt containing S*", etc.; based on this, personalized products containing animal objects can be diffused based on the animal objects in the sample image.

[0153] In the embodiment of the present application, based on two optimization points in the training process of the image generation model (the first optimization point is that during the first training process of the image generation model, the initial representation information of the sample identifier is determined by minimizing the difference between the semantics of the sample image, the semantics of the sample category and the native semantics of the sample identifier, so that the semantics of the sample identifier represented by the determined initial representation information of the sample identifier is close to the semantics of the sample image and the semantics of the sample category, and the initial representation information of the sample identifier is used to participate in the first training of the image generation model, which can quickly guide the image generation model in the appropriate training direction, improve the fitting efficiency of the image generation model, and improve the image generation model. The first optimization point is to improve the generalization of the image generation model; the second optimization point is to train the image generation model based on the difference between the semantics of the sample identifier and the semantics of the sample category, which can improve the similarity between the semantics of the sample identifier and the semantics of the sample category, and associate the semantics of the sample identifier with the semantics of the sample category, so as to better utilize the prior knowledge of the image generation model (that is, the semantics of the sample category). This can reduce the overfitting degree of the image generation model and improve the training effect of the image generation model. It can improve the ability to combine the sample features of the sample image under the sample category with the text semantics in the text description information, thereby improving the image generation effect of the trained image generation model.

[0154] The following is a comparison of the generation effects of the trained image generation model when the optimization points are introduced and when the optimization points are not introduced during the training process of the image generation model.

[0155] like Fig.10 As shown in the figure, the first column of images are sample images, the sample category to which the sample images belong is "hamster", the sample identifier set for the sample images is "[hamster*]", and the text description information containing the sample identifier is "[hamster*] is in the bucket"; the second column is the generation effect of the trained image generation model when the first optimization point is not introduced in the training process of the image generation model; the third column is the generation effect of the trained image generation model when the first optimization point is introduced in the training process of the image generation model. It can be seen that the image generation effect of the trained image generation model when the first optimization point is not introduced is significantly worse than the image generation effect of the trained image generation model when the first optimization point is introduced. By introducing the first optimization point, the image generation model can be quickly guided in the appropriate training direction, the fitting efficiency of the image generation model can be improved, and the generalization of the image generation model can be improved.

[0156] like Fig.11As shown in the figure, the first column of images are sample images, the sample category to which the sample images belong is "dog", the sample identifier set for the sample images is "[dog*]", and the text description information containing the sample identifier is "a painting containing [dog*] wearing a hat"; the second column is the generation effect of the trained image generation model when the second optimization point is not introduced in the training process of the image generation model; the third column is the generation effect of the trained image generation model when the second optimization point is introduced in the training process of the image generation model. It can be seen that the image generation effect of the trained image generation model when the second optimization point is not introduced is significantly worse than the image generation effect of the trained image generation model when the second optimization point is introduced. By introducing the second optimization point, the similarity between the semantics of the sample identifier and the semantics of the sample category can be improved, the semantics of the sample identifier and the semantics of the sample category can be associated, and the prior knowledge of the image generation model (i.e., the semantics of the sample category) can be better utilized. In this way, the overfitting degree of the image generation model can be reduced, so that the trained image processing model can combine the sample features of the sample image under the sample category with the text semantics of the text description information, and the generated image is more natural and diverse.

[0157] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0158] See also Fig.12 , Fig.12 is a structural diagram of an image processing device provided in an embodiment of the present application, see Fig.12 , the image processing device may include the following units:

[0159] An acquisition unit 1201 is used to acquire sample data for training an image generation model, where the sample data includes a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image;

[0160] The processing unit 1202 is used to call the image generation model to extract the representation information of the sample identification, where the representation information of the sample identification is used to represent the semantics of the sample identification, and the semantics of the sample identification is used to express the sample characteristics of the sample image under the sample category;

[0161] The processing unit 1202 is further used to call the image generation model based on the representation information of the sample identification, predict the noise added to the noise image corresponding to the sample image, and obtain the predicted noise; the noise image is obtained by performing noise processing on the sample image, and the noise added by the noise processing is the label noise;

[0162] The processing unit 1202 is also used to update the model parameters of the image generation model according to the difference between the marked noise and the predicted noise, and the difference between the semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model; the trained image generation model is used to denoise the noisy image based on the text description information containing the sample identifier, and generate a diffusion image that is semantically related to the text description information.

[0163] In one implementation, the training steps of extracting the representation information of the sample identification, predicting the noise added to the noisy image, and updating the model parameters of the image generation model are used to train the image generation model once; the processing unit 1202 is further used to perform the following steps:

[0164] According to the training steps, the image generation model is iteratively trained until the training termination condition is reached, thereby obtaining a trained image generation model;

[0165] The iterative training process of the image generation model takes the initial representation information of the sample identifier as the training starting point. The initial representation information of the sample identifier is determined by minimizing the difference between the semantics of the sample image, the semantics of the sample category and the native semantics of the sample identifier. The training starting point means: taking the initial representation information of the sample identifier as the representation information of the sample identifier and participating in the first training process of the image generation model.

[0166] In one implementation, during the iterative training process of the image generation model, the noise images corresponding to the sample images used in each training process of the image generation model are different; the noise images corresponding to the sample images are different in that: the labeling noise added to the sample images by the noise processing in each training process of the image generation model is different;

[0167] The different labeling noises include any one of the following: the types of labeling noises added in each training process of the image generation model are different, and the types of labeling noises added in each training process of the image generation model are the same but the intensities are different.

[0168] In one implementation, the processing unit 1202 is used to determine the initial representation information of the sample identifier by minimizing the difference between the semantics of the sample image, the semantics of the sample category, and the native semantics of the sample identifier, and is specifically used to perform the following steps:

[0169] Extracting representation information of the sample image, where the representation information of the sample image is used to represent the semantics of the sample image;

[0170] Extracting the representation information of the sample category, the representation information of the sample category is used to represent the semantics of the sample category;

[0171] Averaging the representation information of the sample image and the representation information of the sample category to obtain average representation information;

[0172] Extracting native representation information of the sample identifier, where the native representation information of the sample identifier is used to represent the native semantics of the sample identifier;

[0173] The initial representation information of the sample identifier is determined by minimizing the distance between the native representation information of the sample identifier and the average representation information.

[0174] In one implementation, the processing unit 1202 is used to extract the representation information of the sample image, and is specifically used to perform the following steps:

[0175] Extracting original image representation information of the sample image, where the original image representation information is used to represent the original image semantics of the sample image;

[0176] Extracting background-removed representation information of the background-removed image corresponding to the sample image, where the background-removed representation information is used to represent the semantics of the background-removed image;

[0177] The original image representation information and the background removal representation information are determined as the representation information of the sample image.

[0178] In one implementation, the image generation model includes a text encoder and an image diffusion sub-model, the text encoder is used to extract representation information of sample identification, and the image diffusion sub-model is used to predict noise; a training module is added to the text encoder and the image diffusion sub-model;

[0179] The processing unit 1202 is used to update the model parameters of the image generation model according to the difference between the marked noise and the predicted noise, and the difference between the semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model, and is specifically used to perform the following steps:

[0180] According to the difference between labeling noise and prediction noise, and the difference between the semantics of sample identification and the semantics of sample category, the original model parameters in the text encoder and image diffusion sub-model are kept unchanged, and the model parameters of the training modules added in the text encoder and image diffusion sub-model are updated to train the image generation model.

[0181] In one implementation, predicting the noise added to the noisy image is performed by calling an image diffusion sub-model in the image generation model, the image diffusion sub-model includes N cross-attention modules, any one of the N cross-attention modules is represented as an i-th cross-attention module, N is an integer greater than or equal to 2, and i is a positive integer less than or equal to N;

[0182] The processing unit 1202 is used to call the image generation model based on the representation information of the sample identification, predict the noise added to the noise image corresponding to the sample image, and when the predicted noise is obtained, specifically perform the following steps:

[0183] When i=1, the i-th cross attention module is called to perform correlation calculation on the representation information of the sample identifier and the noise image to obtain the cross attention result of the i-th cross attention module;

[0184] When i>1, the i-th cross attention module is called to perform correlation calculation on the cross attention result of the i-1-th cross attention module and the representation information of the sample identifier to obtain the cross attention result of the i-th cross attention module;

[0185] The cross-attention result of the Nth cross-attention module is determined as the prediction noise.

[0186] In one implementation, the trained image generation model includes a trained text encoder and a trained image diffusion sub-model; the acquisition unit 1201 is further configured to perform the following steps:

[0187] Get text description information including sample identification;

[0188] The processing unit 1202 is further configured to perform the following steps:

[0189] Calling the trained text encoder to extract representation information of the text description information; the representation information of the text description information is used to express the semantics of the text description information;

[0190] The trained image diffusion sub-model is called to denoise the noisy image based on the representation information of the text description information, and a diffusion image corresponding to the sample image is generated. The diffusion image is semantically related to the text description information.

[0191] In one implementation, the noise image is obtained by converting the sample image from the pixel space to the coding space and then performing noise processing in the coding space; the processing unit 1202 is used to call the trained image diffusion sub-model, perform denoising processing on the noise image based on the representation information of the text description information, and generate a diffusion image corresponding to the sample image, specifically for performing the following steps:

[0192] In the coding space, calling the trained image generation model based on the representation information of the text description information, denoising the noisy image, and obtaining the coding information of the diffusion image;

[0193] The coding information of the diffusion image is converted from the coding space to the pixel space to obtain the diffusion image.

[0194] In one implementation, when the sample category is the category to which the sample image as a whole belongs, the sample feature of the sample image under the sample category is the overall image feature of the sample image under the sample category, the semantics of the sample identifier is used to express the overall image feature of the sample image under the sample category, and the diffusion image corresponding to the sample image is an image generated by diffusion based on the overall sample image;

[0195] When the sample category is the category to which the sample object in the sample image belongs, the sample features of the sample image under the sample category are the object features of the sample object in the sample image under the sample category, the semantics of the sample identifier is used to express the object features of the sample object in the sample image under the sample category, and the diffusion image corresponding to the sample image is an image generated by diffusion based on the sample object in the sample image.

[0196] According to another embodiment of the present application, Fig.12 The various units in the image processing device shown can be separately or completely combined into one or several other units to form, or one (some) of the units can also be split into multiple smaller units in function to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the image processing device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0197] According to another embodiment of the present application, the program can be executed by running on a general computing device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 3 or Figure 7 A computer program for each step involved in part or all of the method shown in the figure is used to construct a computer program for each step involved in part or all of the method shown in the figure. Fig.12 The image processing apparatus shown in and the image processing method of the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer-readable storage medium, and loaded into the above-mentioned computing device through the computer-readable storage medium and run therein.

[0198] In an embodiment of the present application, in the process of training an image generation model using a single sample image, a sample identifier can be set for the sample image, and the semantics of the sample identifier can be used to express the sample features of the sample image under the sample category. Based on the difference between the semantics of the sample identifier and the semantics of the sample category, the model parameters of the image generation model are updated, and the image generation model is trained. This can improve the similarity between the semantics of the sample identifier and the semantics of the sample category. In other words, the similarity between the sample features of the sample image under the sample category and the semantics of the sample category can be improved. This is conducive to reducing the overfitting degree of the image generation model and improving the training effect of the image generation model.

[0199] Based on the above method and device embodiments, the present application embodiment provides a computer device. Fig.13 , Fig.13 It is a structural diagram of a computer device provided in an embodiment of the present application. Fig.13 The computer device shown includes at least a processor 1301, an input interface 1302, an output interface 1303, and a computer-readable storage medium 1304. The processor 1301, the input interface 1302, the output interface 1303, and the computer-readable storage medium 1304 may be connected via a bus or other means.

[0200] The computer-readable storage medium 1304 may be stored in a memory of a computer device. The computer-readable storage medium 1304 is used to store a computer program. The computer program includes computer instructions. The processor 1301 is used to execute the computer program stored in the computer-readable storage medium 1304. The processor 1301 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which is suitable for implementing a computer program, and is specifically suitable for loading and executing a computer program to implement a corresponding method flow or a corresponding function.

[0201] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the operating system of the computer device. In addition, a computer program suitable for being loaded and executed by a processor is also stored in the storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (Non-Volatile Memory), such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0202] The computer device may be, for example, Figure 2 The integrated device of the model training device and the model application device in the image processing system shown in the figure can be specifically implemented by the processor 1301 to load and execute the computer program stored in the computer-readable storage medium 1304 to implement the above-mentioned Figure 3 or Figure 7 The corresponding steps in the image processing method shown.

[0203] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the image processing method provided in the above-mentioned various optional manners.

[0204] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Acquire sample data for training an image generation model, the sample data including a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image; Calling the image generation model to extract representation information of the sample identifier, where the representation information of the sample identifier is used to represent the semantics of the sample identifier, and the semantics of the sample identifier is used to express sample features of the sample image under the sample category; Calling the image generation model to predict the noise added to the noise image corresponding to the sample image based on the representation information of the sample identifier to obtain the predicted noise; The noise image is obtained by performing noise processing on the sample image, and the noise added by the noise processing is label noise; According to the difference between the marked noise and the predicted noise, and the difference between the semantics of the sample identifier and the semantics of the sample category, updating the model parameters of the image generation model to train the image generation model; The trained image generation model is used to perform denoising on the noisy image based on the text description information containing the sample identifier, and generate a diffusion image semantically related to the text description information.

2. The method according to claim 1, characterized in that The method further comprises: Iteratively training the image generation model until a training termination condition is reached, thereby obtaining a trained image generation model; The iterative training process of the image generation model takes the initial representation information of the sample identifier as the training starting point; wherein, the initial representation information of the sample identifier is determined by minimizing the difference between the semantics of the sample image, the semantics of the sample category and the native semantics of the sample identifier, and the training starting point means: taking the initial representation information of the sample identifier as the representation information of the sample identifier to participate in the first training process of the image generation model.

3. The method according to claim 2, characterized in that In the iterative training process of the image generation model, the noise images corresponding to the sample images used in each training process of the image generation model are different; the noise images corresponding to the sample images are different means that the label noise added by each training process of the image generation model to perform noise processing on the sample images is different; The different labeling noises include any one of the following: the types of labeling noises added in each training process of the image generation model are different; the types of labeling noises added in each training process of the image generation model are the same but the intensities are different.

4. The method according to claim 2, characterized in that The process of determining initial representation information of the sample identifier by minimizing the difference between the semantics of the sample image, the semantics of the sample category and the native semantics of the sample identifier includes: Extracting representation information of the sample image, where the representation information of the sample image is used to represent the semantics of the sample image; Extracting representation information of the sample category, where the representation information of the sample category is used to represent the semantics of the sample category; Performing average processing on the representation information of the sample image and the representation information of the sample category to obtain average representation information; Extracting native representation information of the sample identifier, where the native representation information of the sample identifier is used to represent native semantics of the sample identifier; The initial characterization information of the sample identification is determined by minimizing the distance between the native characterization information of the sample identification and the average characterization information.

5. The method according to claim 4, characterized in that The extracting the representation information of the sample image includes: Extracting original image representation information of the sample image, where the original image representation information is used to represent the original image semantics of the sample image; Extracting background removal representation information of the background removal image corresponding to the sample image, wherein the background removal representation information is used to represent the semantics of the background removal image; The original image representation information and the background removal representation information are determined as the representation information of the sample image.

6. The method according to claim 1, characterized in that The image generation model includes a text encoder and an image diffusion sub-model, wherein the text encoder is used to extract the representation information of the sample identification, and the image diffusion sub-model is used to predict noise; a training module is added to the text encoder and the image diffusion sub-model; The updating of the model parameters of the image generation model according to the difference between the label noise and the predicted noise, and the difference between the semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model, comprises: According to the difference between the marked noise and the predicted noise, and the difference between the semantics of the sample identifier and the semantics of the sample category, the original model parameters in the text encoder and the image diffusion sub-model are kept unchanged, and the model parameters of the training module added in the text encoder and the image diffusion sub-model are updated to train the image generation model.

7. The method according to claim 1, characterized in that Predicting the noise added to the noise image is performed by calling an image diffusion submodel in the image generation model, wherein the image diffusion submodel includes N cross-attention modules, and any one of the N cross-attention modules is represented as an i-th cross-attention module, where N is an integer greater than or equal to 2, and i is a positive integer less than or equal to N; The calling of the image generation model to predict the noise added to the noise image corresponding to the sample image based on the representation information of the sample identifier to obtain the predicted noise includes: When i=1, the i-th cross attention module is called to perform correlation calculation on the representation information of the sample identifier and the noise image to obtain the cross attention result of the i-th cross attention module; When i>1, the i-th cross attention module is called to perform correlation calculation on the cross attention result of the i-1-th cross attention module and the representation information of the sample identifier to obtain the cross attention result of the i-th cross attention module; The cross-attention result of the Nth cross-attention module is determined as the predicted noise.

8. The method according to claim 1, characterized in that The trained image generation model includes a trained text encoder and a trained image diffusion sub-model; the method further includes: Obtaining text description information including the sample identifier; Calling the trained text encoder to extract representation information of the text description information; the representation information of the text description information is used to express the semantics of the text description information; The trained image diffusion sub-model is called to perform denoising on the noisy image based on the representation information of the text description information, so as to generate a diffusion image corresponding to the sample image, wherein the diffusion image is semantically related to the text description information.

9. The method according to claim 8, characterized in that The noise image is obtained by converting the sample image from a pixel space to a coding space and then performing a noise adding process in the coding space; calling the trained image diffusion sub-model to perform a denoising process on the noise image based on the representation information of the text description information to generate a diffusion image corresponding to the sample image, including: In the coding space, calling the trained image generation model based on the representation information of the text description information, denoising the noisy image, and obtaining the coding information of the diffusion image; The coding information of the diffusion image is converted from the coding space to the pixel space to obtain the diffusion image.

10. The method according to claim 1 or 8, characterized in that When the sample category is the category to which the sample image as a whole belongs, the sample feature of the sample image under the sample category is the overall image feature of the sample image under the sample category, the semantics of the sample identifier is used to express the overall image feature of the sample image under the sample category, and the diffusion image corresponding to the sample image is an image generated by diffusion based on the overall sample image; When the sample category is the category to which the sample object in the sample image belongs, the sample features of the sample image under the sample category are the object features of the sample object in the sample image under the sample category, the semantics of the sample identifier is used to express the object features of the sample object in the sample image under the sample category, and the diffusion image corresponding to the sample image is an image generated by diffusion based on the sample object in the sample image.

11. An image processing device, characterized in that: include: An acquisition unit, configured to acquire sample data for training an image generation model, wherein the sample data includes a sample image, a sample category to which the sample image belongs, and a sample identifier set for the sample image; a processing unit, configured to call the image generation model to extract representation information of the sample identifier, wherein the representation information of the sample identifier is used to represent the semantics of the sample identifier, and the semantics of the sample identifier is used to express sample features of the sample image under the sample category; The processing unit is further used to call the image generation model based on the representation information of the sample identifier to predict the noise added to the noise image corresponding to the sample image to obtain the predicted noise; the noise image is obtained by performing noise addition processing on the sample image, and the noise added by the noise addition processing is the label noise; The processing unit is further used to update the model parameters of the image generation model according to the difference between the marked noise and the predicted noise, and the difference between the semantics of the sample identifier and the semantics of the sample category, so as to train the image generation model; The trained image generation model is used to perform denoising on the noisy image based on the text description information containing the sample identifier, and generate a diffusion image semantically related to the text description information.

12. A computer device, characterized in that: The computer device comprises: a processor suitable for implementing a computer program; A computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by the processor and executing the image processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the image processing method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 10 is implemented.

Images