Image processing method, device, equipment, medium, product

Abstract

The present application provides an image processing method, device, equipment, medium, and product; the method includes: obtaining a first foreground image, a first background image, and a first text, wherein the first foreground image includes a first target object, and the first text is used to indicate that the first target object is embedded in the first background image; generating a first combined feature based on the low-frequency signal features of the first background image and the high-frequency signal features of the random noise image; inputting the first combined feature and the first text feature of the first text into an image generation model for image generation processing to obtain a composite image; the image generation model includes a first bypass network constructed based on the first foreground image. The present application supports limited background adaptive changes of the first background image and joint fine-tuning of the first target object, which can ensure that the background area of ​​the composite image is consistent with the first target object and improve the image synthesis quality.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of artificial intelligence technology, and specifically to an image processing method, device and equipment, medium, and product. Background Art

[0002] With the development of computer technology, related technologies can embed personalized objects under background images. For example, the characters in the background image can be removed, and a target character with the same characteristics (such as posture, orientation, color, style, shadow, etc.) as the removed character in the background image can be produced. The produced target character and the background image from which the character is removed can be manually used for mapping to obtain a composite image. However, this image synthesis process is too cumbersome, and in order to ensure that the background area of ​​the composite image can achieve consistency with the target character, a lot of energy and time are required. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, apparatus and equipment, medium, and product. By supporting limited background adaptive changes of a first background image and joint fine-tuning of a first target object, it is possible to ensure that the background area of ​​a synthesized image is consistent with the first target object, thereby improving image synthesis quality and image synthesis efficiency.

[0004] On the one hand, an embodiment of the present application provides an image processing method, the method comprising:

[0005] Acquire a first foreground image, a first background image, and a first text, wherein the first foreground image includes a first target object, and the first text is used to indicate that the first target object is embedded in the first background image;

[0006] Generate a first combined feature based on the low-frequency signal feature of the first background image and the high-frequency signal feature of the random noise image;

[0007] The first combined feature and the first text feature of the first text are input into an image generation model for image generation processing to obtain a composite image; wherein the image generation model includes a first bypass network constructed based on the first foreground image, and the first bypass network contains feature information of the first target object.

[0008] On the other hand, an embodiment of the present application provides an image processing device, the device comprising:

[0009] An acquisition unit, used to acquire a first foreground image, a first background image and a first text, wherein the first foreground image includes a first target object, and the first text is used to indicate that the first target object is embedded in the first background image;

[0010] A processing unit, configured to generate a first combined feature based on the low-frequency signal feature of the first background image and the high-frequency signal feature of the random noise image;

[0011] The processing unit is also used to input the first combined feature and the first text feature of the first text into the image generation model for image generation processing to obtain a composite image; wherein the image generation model includes a first bypass network constructed based on the first foreground image, and the first bypass network contains feature information of the first target object.

[0012] In one embodiment, the image generation model includes a target diffusion network and a first bypass network; the processing unit is used to input the first combined feature and the first text feature of the first text into the image generation model to perform image generation processing, and when the synthetic image is obtained, the following steps are specifically performed:

[0013] Calling the forward diffusion module in the target diffusion network to perform forward diffusion processing on the first combined feature to obtain a first noise feature;

[0014] Calling the back diffusion modules in the first bypass network and the target diffusion network to perform back diffusion processing on the first noise feature based on the first text feature of the first text to obtain a reference image feature;

[0015] The reference image features are subjected to image decoding processing to obtain a synthetic image.

[0016] In one embodiment, the processing unit is used to call the back diffusion module in the first bypass network and the target diffusion network, and perform back diffusion processing on the first noise feature based on the first text feature of the first text to obtain the reference image feature, specifically performing the following steps:

[0017] Inputting a first text feature of a first text into a first bypass network to obtain a bypass weight parameter;

[0018] Updating the backward diffusion module in the target diffusion network based on the bypass weight parameters;

[0019] The first text feature and the first noise feature are input into the updated back diffusion module, and the updated back diffusion module performs back diffusion processing on the first noise feature based on the first text feature to obtain a reference image feature.

[0020] In one embodiment, the acquisition unit is further configured to perform the following steps:

[0021] Acquire a first training sample and a first initial model, wherein the first training sample includes a first foreground image and a second text, wherein the second text includes description information of a first target object; and the first initial model includes an initial bypass network and an initial diffusion network;

[0022] The processing unit is also used to perform the following steps:

[0023] Inputting a first image feature of a first foreground image and a second text feature of a second text into a first initial model, and predicting a first noise added to the first image feature by the first initial model;

[0024] Based on the difference between the first noise added to the predicted first image feature and the second noise actually added to the first image feature, the initial bypass network is trained to obtain the first bypass network.

[0025] In one embodiment, the processing unit is used to input the first image feature of the first foreground image and the second text feature of the second text into the first initial model, and when the first initial model predicts the first noise added to the first image feature, specifically perform the following steps:

[0026] Calling the forward diffusion module in the initial diffusion network to perform forward diffusion processing on the first image feature to obtain a second noise feature, where the second noise feature is obtained by adding the second noise to the first image feature;

[0027] Calling the back diffusion module in the initial bypass network and the initial diffusion network to perform back diffusion processing on the second noise feature based on the second text feature of the second text;

[0028] A first noise added to a first image feature predicted during a back diffusion process is obtained.

[0029] In one embodiment, the acquisition unit is further configured to perform the following steps:

[0030] Acquire a second training sample and a second initial model, wherein the second training sample includes a second background image, a third text, and an embedded reference image, wherein the third text is used to indicate that a second target object is embedded in the second background image; the second initial model includes an initial diffusion network and a second bypass network constructed based on a second foreground image, wherein the second foreground image includes a second target object;

[0031] The processing unit is also used to perform the following steps:

[0032] Inputting the second combined feature and the third text feature of the third text into the second initial model to perform image generation processing to obtain a predicted embedded image;

[0033] Based on the difference between the predicted embedded image and the embedded reference image, the initial diffusion network is trained to obtain the target diffusion network.

[0034] In one embodiment, when the processing unit is used to generate the first combined feature based on the low-frequency signal feature of the first background image and the high-frequency signal feature of the random noise image, the processing unit specifically performs the following steps:

[0035] Calling a low-pass filter to perform low-pass filtering on the Fourier features of the first background image to obtain low-frequency signal features of the first background image;

[0036] Calling a high-pass filter to perform high-pass filtering on the Fourier characteristics of the random noise image to obtain the high-frequency signal characteristics of the random noise image;

[0037] Performing splicing processing on the low-frequency signal features of the first background image and the high-frequency signal features of the random noise image to obtain spliced ​​signal features;

[0038] Perform inverse Fourier transform on the concatenated signal features to obtain the first combined features.

[0039] In one embodiment, the first text is specifically used to instruct to replace the third target object in the first background image with the first target object; the processing unit is further used to perform the following steps:

[0040] identifying a detection image region containing a third target object from the first background image;

[0041] Performing a cutout process on the detection image area in the first background image to obtain a background template image;

[0042] Perform Fourier transform on the background template image to obtain Fourier features of the first background image.

[0043] On the other hand, an embodiment of the present application provides a computer device, the computer device comprising:

[0044] A processor for loading and executing computer programs;

[0045] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image processing method is implemented.

[0046] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor and executing the above-mentioned image processing method.

[0047] On the other hand, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned image processing method is implemented.

[0048] The technical solution provided by the embodiment of the present application can obtain a first foreground image, a first background image and a first text, wherein the first foreground image contains a first target object, and the first text is used to indicate that the first target object is embedded in the first background image. A first combined feature is generated based on the low-frequency signal features of the first background image and the high-frequency signal features of the random noise image; the low-frequency signal features of the first background image contain background environment information such as large areas of environment and large sections of lines in the first background image, and the high-frequency signal features of the random noise image mainly contain noise. The first combined feature and the first text feature of the first text are input into the image generation model for image generation processing to obtain a composite image. On the one hand, since the low-frequency signal features will exist in the information flow of the image generation model (referring to the information transmission and aggregation process within the model), it can be ensured that the background environment information such as large areas of environment and large lines in the first background image can be retained in the synthetic image. At the same time, the high-frequency signal features that determine more picture details in the synthetic image can be adjusted with the help of a control signal (i.e., the first text feature of the first text) to support the limited background adaptive change of the first background image; on the other hand, the image generation model includes a first bypass network constructed based on the first foreground image, and the first bypass network contains feature information of the first target object. Therefore, the image generation model can support joint fine-tuning of the first target object, that is, with the help of the feature information of the first target object, the high-frequency signal features that determine the picture details are more accurately adjusted, so that the background area of ​​the synthetic image can maintain a consistent effect with the first target object, and finally a synthetic image with coordinated effects is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a schematic diagram of the structure of an initial diffusion network provided in an embodiment of the present application;

[0050] Figure 2 This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0051] Figure 3 This is a schematic diagram of an image processing method provided in an embodiment of the present application. Figure 1 ;

[0052] Figure 4 is a structural schematic diagram of a first initial model provided in an embodiment of the present application;

[0053] Figure 5 It is a flowchart of a text encoding process provided by an embodiment of the present application;

[0054] Figure 6 This is a schematic diagram of an image processing method provided in an embodiment of the present application. Figure 2 ;

[0055] Figure 7is a structural schematic diagram of a second initial model provided in an embodiment of the present application;

[0056] Figure 8 is a schematic diagram of a process for generating a second combination feature provided by an embodiment of the present application;

[0057] Fig. 9 This is a schematic diagram of an image processing method provided in an embodiment of the present application. Figure 3 ;

[0058] Fig.10a This is a schematic diagram of an image processing method provided in an embodiment of the present application. Figure 4 ;

[0059] Fig.10b is a schematic diagram of the effect of a synthetic image provided by an embodiment of the present application;

[0060] Fig.11 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0061] Fig.12 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0063] In an embodiment of the present application, an image processing scheme is proposed, which can embed a personalized object into a specified background image to obtain a composite image, and ensure that the background area of ​​the composite image maintains a consistency effect with the personalized object. The so-called consistency effect refers to the fact that the personalized object and the background area in the composite image have the same characteristics or style, and visually appear harmonious and unified, without obvious inconsistency or illogicality. A personalized object is an object specified by a user according to his or her specific needs. This application does not limit the type of personalized object. For example, in social media, a user expects to freely replace the characters in the background image with himself or herself, then the personalized object can be the user's own face. In e-commerce, a merchant expects to generate a product display image under a specified background, then the personalized object can be a product. The following takes the personalized object as the first target object and the specified background image as the first background image as an example to illustrate the image processing scheme provided by the present application.

[0064] In order to facilitate understanding of the image processing solution provided by the present application, the image generation model designed by the present application is briefly introduced. Exemplarily, the image generation model may include a target diffusion network and a first bypass network, and the image generation model has the ability to embed the first target object into any background image (such as the first background image).

[0065] 1. Target diffusion network:

[0066] The initial diffusion network is a trained text-to-image generation model that can generate high-quality, high-resolution, and high-realistic images based on any text input. For example, the initial diffusion network can be a trained stable diffusion (SD) model.

[0067] For example, see Figure 1 The structural schematic diagram of the initial diffusion network shown in FIG. 1 , the initial diffusion network may include an image encoding module, a forward diffusion module, a backward diffusion module, a text encoding module, and an image decoding module.

[0068] The image coding module is used to perform image coding processing on any acquired image X to map the image X from the pixel space to the latent space to obtain the corresponding image feature Z (which can be a vector representation); illustratively, the image coding network can use a variational auto encoder (VAE), which can map any image from the pixel space to the latent space to capture the essential information of the image. Among them, the pixel space refers to the representation of the image at the pixel level, that is, the image is represented as a pixel matrix, and the value of each pixel in the pixel matrix represents the color or grayscale value of the corresponding position on the image. The latent space refers to the low-dimensional vector space obtained after the image is encoded by a specific model (such as VAE). In the image coding task, the latent space can be regarded as an abstract representation, in which each dimension represents the characteristics or attributes of the image. By mapping the image from the pixel space to the latent space, the image coding module can extract the essential information of the image and retain a certain reconstruction capability. For example, the vector in the latent space can be restored to the image through image decoding processing. The so-called essential information of an image refers to the most important and representative features or content in the image. It contains the main visual information conveyed by the image. This essential information can be the visual attributes of the image, such as shape, texture, color, object, scene, etc.

[0069] The forward diffusion module can also be called the noise addition module, which is used to perform forward diffusion processing (i.e., noise addition operation) on the image feature Z to introduce randomness and diversity of noise into the image feature Z and obtain the corresponding noise feature Z. TIn the process of forward diffusion, T (positive integer) noises (which can be any noise such as Gaussian noise, uniform noise, etc.) are added to the image feature Z. T Represents the noise features corresponding to time step T (i.e., the image features after adding noise for the Tth time).

[0070] The back diffusion module is used to analyze the noise characteristics Z T Back diffusion is performed to obtain the noise characteristic Z T Predict the noise added to the image feature Z, and then in the noise feature Z T The noise added in the predicted image feature Z is removed to obtain the restored image feature Z (i.e., image feature Z'). In a specific implementation, the back diffusion module can be T cascaded denoising modules. For example, the denoising module can be a U-Net (U-shaped Convolutional Neural Network). In the back diffusion process (referred to as the back diffusion process), the first denoising module can receive the noise feature Z T , and based on the noise characteristic Z T Predict the noise added in the Tth noise addition operation, in the noise feature Z T Remove the predicted noise added in the Tth noise addition operation to obtain the restored noise feature Z T-1 (i.e., the image features after adding noise for the T-1th time), the second denoising module can receive the noise feature Z T-1 , and based on the noise characteristic Z T-1 Predict the noise added in the T-1th noise addition operation, in the noise feature Z T-1 Remove the predicted noise added in the T-1th noise adding operation to obtain the restored noise feature Z T-2 (i.e., the image feature after the T-2th noise addition), and so on, finally the Tth denoising module can receive the noise feature Z1, and predict the noise added in the first noise addition operation based on the noise feature Z1, and remove the predicted noise added in the first noise addition operation from the noise feature Z1 to obtain the image feature Z'.

[0071] The text encoding module can be used to perform text encoding processing on any text to obtain corresponding text features. For example, the text encoding module can be a text encoder of CLIP (Contrastive Language-Image Pre-training, a cross-modal self-supervised learning framework), which is obtained by performing a contrastive learning task between text and image.

[0072] Among them, the text features output by the text encoding module will be input into the back diffusion module as a control signal, so that the back diffusion module can flexibly adjust the back diffusion process (i.e., the process of predicting noise) based on the content of the text, so that the generated image output by the initial diffusion network can match the description of the text, improve the accuracy and semantic relevance of the image generation, and make the generated image more in line with user needs. For example, if the text includes "eyes", an image containing eyes will be generated. Therefore, the image feature Z (i.e., image feature Z') restored by the back diffusion module does not emphasize that it must be consistent with the real image feature Z. Exemplarily, the back diffusion module can be T cascaded denoising modules, and the denoising module can include an encoder and a decoder, and the encoder and the decoder are connected by jumps to fuse information at different scales; the encoder usually includes multiple downsampling layers for gradually reducing the spatial resolution of the feature; the decoder usually includes multiple upsampling layers for gradually restoring the features output by the encoder to the original spatial resolution. Each sampling layer (including upsampling layer and downsampling layer) in the denoising module can be an attention layer (i.e., QKV layer) that introduces a cross-attention mechanism, so that the features of the back-diffusion module can be cross-attended together with the text features, so that the text features can be input into the back-diffusion module as a control signal. Among them, the features of the back-diffusion module include any one or more of the following: the noise features received by the back-diffusion module (such as the noise feature Z T ) and the features generated by the back diffusion module (such as the features output by a layer in the first denoising module, the noise feature Z predicted by the first denoising module T-1 ).

[0073] The cross attention mechanism is an attention mechanism used to establish associations and interactions between different data. Specifically, in the cross attention mechanism, a sequence (i.e., sequence 1, such as noise feature Z T) Through cross-attention learning with another sequence (i.e., sequence 2, such as text features), a dependency relationship between the two different sequences is established. The dependency relationship refers to the degree of attention of each element in sequence 1 to each element in sequence 2. The execution process of cross-attention learning is roughly as follows: ① Define query elements (Q), key elements (K), and value elements (V): map sequence 1 to query elements, and map sequence 2 to both key elements and value elements; ② Calculate attention weights: perform dot product operations (or other appropriate similarity measures) on query elements and key elements, and perform normalization on the results of the dot product operations through the softmax function to obtain a set of attention weights. This step is to compare each element of sequence 1 with each element of sequence 2 to determine the degree of attention of each element in sequence 1 to each element in sequence 2. ③ Calculate output: Use the obtained attention weights to perform weighted summation on the value elements to obtain the feature representation of sequence 1 after attention focus.

[0074] The image decoding module is used to perform image decoding processing on the image feature Z' output by the back diffusion module to map the image feature Z' from the latent space back to the pixel space to obtain the generated image Y.

[0075] It can be seen that the initial diffusion network can generate noise features in the latent space through the forward diffusion module in the initial diffusion network, and can also convert the input text into a control signal to guide the backward diffusion module in the initial diffusion network to perform the backward diffusion process of the noise features, and then use the image decoding module in the initial diffusion network to generate a generated image that conforms to the text description.

[0076] Based on the fact that the initial diffusion network has the ability of text and image generation, this application further trains the initial diffusion network to obtain a target diffusion network (i.e., the initial diffusion network after training). Among them, during the training process, the initial diffusion network will learn how to naturally merge the personalized object with the specified background image (i.e., learn how to maintain the consistency effect). ① In order to make the target diffusion network have a better image synthesis effect, this application will use the low-frequency signal features of the background image and the high-frequency signal features of the random noise image to generate a combined feature, and use the combined feature as the input of the forward diffusion module in the initial diffusion network, so that the low-frequency signal features in the combined feature can exist in the information flow of the initial diffusion network, so that the initial diffusion network can learn how to retain the background environment information such as large areas of environment and large sections of lines in the background image in the synthetic image during the training process, and learn how to adjust the high-frequency signal features in the combined feature based on the control signal (i.e., text feature) so that the picture details in the synthetic image can be more accurate. In this way, when the target diffusion network is actually applied, it can directly retain the background environment information such as large areas of environment and large sections of lines in the background image in the synthetic image, and realize the precise fine-tuning of the picture details to generate a synthetic image with coordinated effects. ② In order to improve the generalization ability of the target diffusion network, the learning ability of the initial diffusion network for the fusion of personalized objects and background images can be strengthened by introducing diverse sample data. For example, the initial diffusion network can learn to embed object 1 into image 1, image 2, and image 3, embed object 2 into image 1 and image 4, and embed object 3 into image 5 and image 6. In this way, the target diffusion network can achieve good performance when embedding different personalized objects into different background images. Even when the target diffusion network is applied to the fusion of new personalized objects and background images, it still has good performance.

[0077] In order to fuse personalized objects with background images, the target diffusion network needs to master the characteristic information of personalized objects. The characteristic information of personalized objects refers to information that can describe personalized objects, such as characteristic information such as the shape and color of a water cup, and characteristic information such as the shape of eyes, nose, and mouth on a person's face, which is conducive to the target diffusion network to generate more stable (i.e., it will not become other objects in the synthetic image, or will not be similar to the personalized object) and accurate results; for example, if you want to embed a water cup into the background image, but the styles of water cups are diverse, if the target diffusion network does not know the style of water cup that the user expects to embed, it is very likely to generate a synthetic image that does not meet the requirements. Therefore, the embodiment of the present application will train a bypass network of personalized objects, and the bypass network of personalized objects contains characteristic information of personalized objects. The bypass network of personalized objects is usually parallel to the target diffusion network, and is used to provide the characteristic information of personalized objects to the target diffusion network, so as to help the target diffusion network model more accurately embed personalized objects on the specified background image, which is conducive to improving the accuracy and realism of personalized objects in the synthetic image, making the synthetic image more realistic and natural. When the initial diffusion network learns to naturally merge the personalized object with the specified background image, the bypass network of the personalized object will also be trained in parallel with the initial diffusion network with the help of the feature information of the personalized object provided by the bypass network of the personalized object.

[0078] 2. The first bypass network:

[0079] The first bypass network refers to the bypass network of the first target object, that is, the present application will train a dedicated bypass network for each personalized object, for example, object 1 has bypass network 1, object 2 has bypass network 2, and object 3 has bypass network 3. Exemplarily, in order to train the first bypass network, a first foreground image may be obtained, and the first foreground image refers to an image containing the first target object, and then the initial bypass network is trained using the first foreground image to obtain the first bypass network. During the training process, the initial bypass network can learn the feature information of the first target object based on the first target object in the first foreground image, so that the first bypass network can use the feature information of the first target object to assist the target diffusion network in better embedding and fine-tuning the first target object when generating a synthetic image, thereby achieving a more personalized and accurate generation effect.

[0080] The following briefly describes the application scenarios to which the image processing solution provided by this application is applicable. It should be noted that the following scenarios are only used to illustrate the embodiments of this application and are not limiting. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs. Figure 2 , Figure 22 is a schematic diagram of an application scenario provided by an embodiment of the present application. The application scenario includes a terminal device 201 and a server 202, and the terminal device 201 and the server 202 can communicate with each other through a communication network.

[0081] In an optional implementation, the communication network may be a wired network or a wireless network. Therefore, the terminal device 201 and the server 202 may be directly or indirectly connected via a wired or wireless communication. For example, the terminal device 201 may be indirectly connected to the server 202 via a wireless access point, or the terminal device 201 may be directly connected to the server 202 via the Internet, and this application does not limit this.

[0082] The terminal device 201 includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc. The server 202 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc.

[0083] In one implementation, various clients may be installed on the terminal device 201, and the client may support text input and image input (can be used to receive text, a specified background image, and a foreground image containing a personalized object), and the client may be an application that supports image generation. For example, the client is an intelligent creation system, and the server 202 is a background server of the intelligent creation system. The server 202 may deploy an image generation model to provide image synthesis services for the intelligent creation system through the image generation model.

[0084] If the personalized object that the user wants to embed has its bypass network trained by the server 202, then the client in the terminal device 201 does not actually need to upload the foreground image containing the personalized object. The server 202 can directly use the target diffusion network and the bypass network of the existing personalized object to build an image generation model, and use the image generation model to provide image synthesis services. For example, the text uploaded by the client may contain a text description of the personalized object (or a preset identifier for uniquely identifying the personalized object), and the text description of the personalized object (or the preset identifier for uniquely identifying the personalized object) can be used to search whether there is a bypass network of the corresponding personalized object. When there is a bypass network of the corresponding personalized object, the target diffusion network and the bypass network of the personalized object can be directly used to build an image generation model. However, if the server 202 does not contain the bypass network of the personalized object that the user wants to embed, the client in the terminal device 201 needs to upload a foreground image containing the personalized object to train the bypass network of the personalized object using the foreground image, and then use the target diffusion network and the bypass network of the personalized object to form an image generation model. The embodiment of the present application is described by taking the case where the client uploads a foreground image containing a personalized object as an example.

[0085] Figure 2 The figure is only an example. In fact, the number of terminal devices 201 and servers 202 is not limited. For example, the number of the terminal device 201 can be only one, or the number of the terminal devices 201 can be dozens or hundreds, or more.

[0086] The collection and processing of relevant data in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations. The acquisition of personal information requires the knowledge or consent of the individual subject (or the legal basis for obtaining the information), and the subsequent use and processing of data shall be carried out within the scope of authorization of laws and regulations and the subject of personal information. For example, when the embodiments of this application are applied to specific products or technologies, such as obtaining background images and foreground images containing personalized objects, the permission or consent of the creators of the background images and foreground images shall be obtained, and the collection, use and processing of relevant data shall comply with the relevant laws, regulations and standards of the relevant regions.

[0087] The following describes specific embodiments of the image processing solution provided in this application.

[0088] See also Figure 3 , Figure 3 This is a schematic diagram of an image processing method provided in an embodiment of the present application. Figure 1 The image processing method introduces the process of constructing the first bypass network, and the image processing method can be performed by a model training device, such as Figure 2The image processing method mainly includes but is not limited to the following steps S301 to S303:

[0089] S301, obtaining a first training sample and a first initial model, wherein the first training sample includes a first foreground image and a second text, wherein the second text includes description information of a first target object; and the first initial model includes an initial bypass network and an initial diffusion network.

[0090] The first target object is an object to be embedded in a specified background image (such as the first background image). The first target object can be specified by the user according to his or her specific needs. For example, the first target object can be an object, a person, an animal, a plant, etc. This application does not limit the type of the first target object.

[0091] The first foreground image includes a first target object, and the first foreground image is used to construct a first bypass network. The first bypass network only needs to learn feature information of the first target object. Therefore, the first foreground image does not need to include other objects except the first target object, and there is no restriction on the background of the first target object in the first foreground image.

[0092] The second text is a text used to describe the first foreground image. The second text contains description information of the first target object. The description information of the first target object can be the name of the first target object in text form. For example, if the first foreground image is an image containing an alarm clock, then the second text is: "This is a photo of an alarm clock", and the description information of the first target object is: "Alarm clock". In order to distinguish the first target object from other objects, the present application will use the first preset identifier to replace the description information of the first target object in the second text. The first preset identifier is only used to identify the first target object. For example, the first preset identifier is , replace "alarm clock" in the second text: "This is a photo of an alarm clock" with , you can get the second text after replacement: "This is a The alarm clock has various styles, but the first preset identifier is used. It is possible to identify only the alarm clock in the first foreground image.

[0093] In one implementation, a first training sample and a first initial model may be obtained, the first initial model includes an initial bypass network and an initial diffusion network, and the first training sample is a sample used to train the initial bypass network in the first initial model, that is, during the training process of the first initial model, the present application will only adjust the network parameters of the initial bypass network in the first initial model, and will not adjust the network parameters of the initial diffusion network, because only the initial bypass network needs to learn the feature information of the first target object. The first training sample includes a first foreground image and a second text; illustratively, the first foreground image may be provided when the user wishes to embed the first target object into a specified background image, and the second text may be obtained by invoking an image processing model with image understanding capabilities to process the first foreground image.

[0094] For example, see Figure 4 The structural diagram of the first initial model shown in FIG. 1 is that the backward diffusion module in the initial diffusion network can be T cascaded noise reduction modules, each of which can include multiple sampling layers (i.e., QKV layers), and the initial bypass network can include multiple bypass modules (i.e., layer), a sampling layer in the denoising module (i.e., QKV layer) can correspond to a bypass module in the initial bypass network (i.e., layer), the bypass module in the initial bypass network (i.e. The layer) can have the same structure and network parameters as the corresponding sampling layer (i.e., the QKV layer) (i.e., the bypass module is initialized directly using the corresponding sampling layer). Figure 4 This is only an example of the first initial model. In practice, it is not necessary to insist that each sampling layer must correspond to a bypass module.

[0095] S302: Input a first image feature of a first foreground image and a second text feature of a second text into a first initial model, and use the first initial model to predict a first noise added to the first image feature.

[0096] The first image feature is used to reflect the essential information of the first foreground image, that is, the first image feature can be used to reflect the most important and representative content (i.e., the first target object) in the first foreground image. In one implementation, the image encoding module in the initial diffusion network can be called to perform image encoding processing on the first foreground image to obtain the first image feature. In another implementation, the object encoding model can be used to perform image encoding processing on the first foreground image to obtain the first image feature. The object encoding model can identify the first target object from the first foreground image, and then focus more on image encoding processing of the first target object. The first image feature obtained in this way is conducive to strengthening the feature information of the first target object.

[0097] The second text feature can be used to describe the semantics of the second text, and the second text feature can be obtained by performing text encoding processing on the replaced second text. In one implementation manner, the text encoding module in the initial diffusion network can be called to perform text encoding processing on the replaced second text to obtain the second text feature. In another implementation manner, the replaced second text can be segmented to obtain a segmentation sequence, where the segmentation sequence contains at least one segment, and the at least one segment contains a first preset identifier. A segment is the smallest semantic unit of text processing. Each segment in the segmentation sequence can be mapped to a word embedding vector respectively through a word embedding layer to obtain the second text feature. Exemplarily, referring to the schematic flow diagram of text encoding processing as shown in Figure 5 The replaced second text: "This is a photo" can be input into a tokenizer. The tokenizer performs segmentation processing on the replaced second text and looks up the indexes of the obtained segments from a preset segmentation index table. For example, the index corresponding to the segment "This is" is 508, the index corresponding to the segment "a kind of" is 641, the descriptive word " " corresponds to the index , the index indicates that for an unknown segment, an index will be temporarily assigned. The index corresponding to the descriptive word "of" is 73, and the index corresponding to the descriptive word "photo" is 701. Each segment index is subjected to word embedding processing through a word embedding layer, and the index corresponding to each segment is mapped from a numerical space (a set containing numbers) to a vector space (a set containing vectors) to obtain the word embedding features of each segment. The word embedding layer maps words to a low-dimensional real vector space through word embedding technology, so that words with similar meanings are closer in this vector space, thereby better capturing the semantic information of words. The word embedding layer contains word embedding features corresponding to a large number of words. For example, Figure 5 the word embedding layer in 8000 contains word embedding features corresponding to 8000 words respectively: E0, E1, E2,..., E 8000 . Since the word embedding feature corresponding to the segment "This is" is E 508 , the index corresponding to the segment "a kind of" is E 641 , the word embedding feature corresponding to the descriptive word " " is (generated by random initialization and will be gradually improved as the word embedding layer is trained), the word embedding feature corresponding to the descriptive word "of" is E 73 , and the word embedding feature corresponding to the descriptive word "photo" is E 701Therefore, the word embedding features of each word segment can be combined in the order of each word segment to obtain the second text feature, or the word embedding features of each word segment can be input into the attention calculation module for attention calculation to obtain the second text feature. Regardless of which of these two methods is used, the network parameters need to be adjusted. That is, when adjusting the network parameters of the initial diffusion network, the network parameters of the text encoding module or the word embedding layer also need to be adjusted, so that the text encoding module or the word embedding layer can learn the first preset identifier (such as ) semantics and context, in the model application stage, the features of the first preset identifier output by the text encoding module or word embedding layer (such as ) can bind the feature information of the first target object, which is conducive to making the image synthesis result more stable.

[0098] In one embodiment, the first image feature of the first foreground image and the second text feature of the second text may be input into the first initial model, and the first initial model may predict the first noise added to the first image feature; this embodiment may specifically include the following steps 11 to 13:

[0099] Step 11, calling the forward diffusion module in the initial diffusion network, performing forward diffusion processing on the first image feature, and obtaining a second noise feature, where the second noise feature is obtained by adding the second noise to the first image feature. Exemplarily, the second noise feature can be obtained by performing T noise addition operations on the first image feature through the forward diffusion module in the initial diffusion network, and the second noise can be the total noise actually added by the T noise addition operations.

[0100] Step 12: Call the backward diffusion module in the initial bypass network and the initial diffusion network to perform backward diffusion processing on the second noise feature based on the second text feature of the second text.

[0101] In one embodiment, the second text feature of the second text can be input into the initial bypass network to obtain an initial bypass weight parameter; specifically, each bypass module in the initial bypass network will output an initial bypass weight parameter. The backward diffusion module in the initial diffusion network is updated based on the initial bypass weight parameter. The second text feature and the second noise feature are input into the updated backward diffusion module, and the updated backward diffusion module performs backward diffusion processing on the second noise feature based on the second text feature. The initial bypass network can be a fine-tuning network generated by a fine-tuning technology (LoRA technology) based on a deep learning model. The initial bypass network can learn the feature information of the first target object through the first target object in the second background image.

[0102] Among them, the deep learning model is a machine learning model based on artificial neural networks. It has strong capabilities in processing complex data and learning complex patterns. It is usually composed of a multi-layer neural network, where each layer contains multiple neurons, and each layer is connected by weights for information transmission and feature extraction. LoRA (Low-Rank Adaptation) technology is a technology for fine-tuning deep learning models. It freezes most of the parameters of the pre-trained model and introduces a trainable low-rank matrix to adapt to specific downstream tasks.

[0103] For example, assume that any sampling layer in the back diffusion module is denoted as sampling layer i (such as QKV i ), sampling layer i (such as QKV i ) corresponds to bypass module i (i.e. ), i is a positive integer. The second text feature can be input into the bypass module i (i.e. ), by bypass module i (i.e. ) maps the second text feature to Q, K, and V respectively for cross-attention learning, and obtains the bypass module i (i.e. ) output: (i.e. the initial bypass weight parameter output by bypass module i), bypass module i (i.e. ) output: It can be used to reflect the learned feature information of the first target object. In addition, if sampling layer i is not the first sampling layer, the output out_QKV of sampling layer i-1 (i.e., the first sampling layer before sampling layer i) can be i-1 and the second text feature input sampling layer i (i.e. QKV i ), by sampling layer i (i.e. QKV i ) Map the second text features to K and V respectively, and convert the output out_QKV of the sampling layer i-1 i-1 Mapped to Q for cross-attention learning, we get sampling layer i (i.e., QKV i ) output: out_QKV i If sampling layer i is the first sampling layer, the second noise feature and the second text feature can be input into sampling layer i (i.e., QKV j ), by sampling layer i (i.e. QKV i ) The second text feature is mapped to K and V respectively, and the second noise feature is mapped to Q for cross-attention learning to obtain the sampling layer i (i.e., QKV i ) output: out_QKV i Then, the sampling layer i (i.e., QKV i ) output: out_QKV i and bypass module i (i.e. ) output: Perform weighted fusion to obtain an input in_QKV of sampling layer i+1 (i.e., the first sampling layer after sampling layer i) i+1 , for example, the input in_QKV of sampling layer i+1 i+1 for: (i.e., the process of updating the back-diffusion module with the initial bypass weight parameters and performing the back-diffusion processing with the updated back-diffusion module), α is greater than 0, and α is less than or equal to 1. The input of the sampling layer i+1 is in addition to in_QKV i+1 In addition, it also contains the second text feature, that is, for the sampling layer i+1, in_QKV i+1 is taken as the output of sampling layer i.

[0104] In a feasible implementation, the second text feature and the second noise feature can be input into the bypass module i (i.e. ), by bypass module i (i.e. ) The second text feature is mapped to K and V respectively, and the second noise feature is mapped to Q to perform cross-attention learning, and the bypass module i (i.e. ) output: (i.e., the initial bypass weight parameter output by bypass module i).

[0105] Step 13, obtaining the first noise added to the first image feature predicted during the backward diffusion process. The first noise includes: the noise added to the first image feature during the T-th noise addition operation predicted by the first noise reduction module in the backward diffusion module, the noise added to the first image feature during the T-1-th noise addition operation predicted by the second noise reduction module in the backward diffusion module, ..., the noise added to the first image feature during the 1st noise addition operation predicted by the T-th noise reduction module in the backward diffusion module. The first noise can be the total noise added by the predicted T noise addition operations.

[0106] S303: Based on the difference between the first noise added to the predicted first image feature and the second noise actually added to the first image feature, the initial bypass network is trained to obtain a first bypass network.

[0107] In the embodiment of the present application, a first loss parameter can be generated according to the difference between the first noise added to the predicted first image feature and the second noise actually added to the first image feature; the larger the difference, the larger the first loss parameter, and the smaller the difference, the smaller the first loss parameter. Then, stochastic gradient descent (SGD) or other optimization algorithms can be used to optimize (i.e., adjust) the network parameters of the initial bypass network (and the text encoding module or word embedding layer) in the direction of reducing the first loss parameter to implement the training process of the initial bypass network (and the text encoding module or word embedding layer). The "in the direction of reducing the first loss parameter" mentioned here refers to: the optimization direction with the goal of minimizing the first loss parameter; by optimizing in this direction, the first loss parameter generated again by the first initial model after each optimization of the initial bypass network (and the text encoding module or word embedding layer) must be less than the first loss parameter generated by the first initial model before the optimization of the initial bypass network.

[0108] In the actual training process of the initial bypass network (and the text encoding module or word embedding layer), it is necessary to continuously obtain the first training sample to train the initial bypass network (and the text encoding module or word embedding layer). Each time the training is performed, the network parameters of the initial bypass network (and the text encoding module or word embedding layer) are updated once (i.e., adjusted once). If the first training stop condition is reached after multiple updates, it can be determined that the training process of the initial bypass network (and the text encoding module or word embedding layer) is completed, and the initial bypass network (and the text encoding module or word embedding layer) obtained by the last training can be determined as the target bypass network (and the trained text encoding module or the trained word embedding layer). The first training stop condition includes but is not limited to at least one of the following conditions: the first loss parameter is less than the set loss threshold, and the number of training times reaches the first preset number of times.

[0109] In a feasible implementation, a first training set can be obtained, the first training set includes multiple first training samples, the first training set can be divided into multiple first training batches, each first training batch can include one or more first training samples, and one first training batch can train the initial bypass network (and the text encoding module or word embedding layer) once, and the first training set can be iterated a first preset number of times in total (that is, each first training sample in the first training set is trained once).

[0110] It can be seen that the embodiment of the present application can input the first foreground image and the second text into the first initial model, so that the initial bypass network (and the text encoding module or the word embedding layer) in the first initial model can better understand the feature information of the first target object during the training process, and then gradually bind the feature information of the first target object to the first bypass network (and the features of the first preset identifier generated by the trained text encoding module or the trained word embedding layer), which is beneficial to improving the effectiveness of image content constraints based on control signals and maintaining the stability of image synthesis results.

[0111] See also Figure 6 , Figure 6 This is a schematic diagram of an image processing method provided in an embodiment of the present application. Figure 2 The image processing method introduces the process of constructing a target diffusion network, and the image processing method can be performed by a model training device, such as Figure 2 The image processing method mainly includes but is not limited to the following steps S601 to S604:

[0112] S601, obtaining a second training sample and a second initial model, wherein the second training sample includes a second background image, a third text and an embedded reference image, wherein the third text is used to indicate that a second target object is embedded in the second background image; the second initial model includes an initial diffusion network and a second bypass network constructed based on a second foreground image, and the second foreground image includes a second target object.

[0113] The second training sample includes a second background image, a third text, and an embedded reference image. The second background image can be any image. The third text is used to indicate that the second target object is embedded in the second background image, and the second target object can be any object. When the third text contains description information of the second target object, the description information of the second target object needs to be replaced with a second preset identifier, which is a preset identifier used only to identify the second target object. In addition to indicating that the second target object is embedded in the second background image, the third text can also indicate other image synthesis constraints. For example, the second target object can be the first target object, and the second preset identifier can be When performing image synthesis constraints, the third text can be: "Replace the xx object in the second background image with ", the third text can be " The third text may also indicate other image content constraints. For example, the third text may be "add Embedded into the second background image and converted to anime style", the third text can be " Embedded into the second background image, and Wearing sunglasses". The embedded reference image is an image that conforms to the description of the third text. This application uses the embedded reference image as a benchmark so that the predicted embedded image generated by the second initial model can be as close to the embedded reference image as possible. Exemplarily, the second background image and the embedded reference image can be collected from game scenes, film and television drama scenes.

[0114] The second training sample is a sample for training the initial diffusion network in the second initial model. That is to say, during the training process of the second initial model, the present application will only adjust the network parameters of the initial diffusion network in the second initial model, and will not adjust the network parameters of the second bypass network, because the second bypass network is a trained network, and the second bypass network contains the feature information of the second target object, and the second bypass network can accurately provide the feature information of the second target object to the initial diffusion network. The training logic of the second bypass network is consistent with the training logic of the first bypass network. For example, the second target object can be the first target object, and the second foreground image can be the first foreground image. In this way, by executing the above Figure 3 Steps S301-S303 in the training are performed to obtain the second bypass network.

[0115] In one implementation, when the third text is used to indicate that the fourth target object (such as the xx object) in the second background image is replaced with the second target object, it is necessary to identify the detection image area containing the fourth target object from the second background image, and perform a subtraction process on the detection image area in the second background image to obtain the subtracted second background image. For example, the second background image is detected for objects or people, a frame diagram is taken, and the pixels in the frame diagram are set to 0. For another example, a method of object or person segmentation is adopted, and pixels of a preset ratio (such as 10% of the length and width of the external matrix of the segmentation area) are expanded outward from the segmentation edge for subtraction. Subsequent operations on the second background image are replaced with operations on the subtracted second background image, for example, image coding is performed on the second background image to obtain the image features of the second background image, and replaced with image coding is performed on the subtracted second background image to obtain the image features of the second background image.

[0116] For example, see Figure 7 The structural diagram of the second initial model shown in FIG. 1 shows that the second initial model includes an initial diffusion network and a second bypass network. The initial diffusion network includes an image encoding module, an image decoding module, a forward diffusion module, and a backward diffusion module (i.e. Figure 7 T noise reduction modules in the second bypass network), each bypass module in the second bypass network (i.e. The layer) is in parallel with the corresponding sampling layer (i.e., QKV layer) in the back diffusion module. Figure 3It can be seen from the corresponding embodiment that the present application will train the text encoding module (or word embedding layer) so that the trained text encoding module (or trained word embedding layer) can bind the feature information of the first target object to the features of the first preset identifier. Therefore, the present application can also train the text encoding module (or trained word embedding layer) so that the trained text encoding module (or trained word embedding layer) can bind the feature information of the second target object to the features of the second preset identifier. The second initial module can also include the trained text encoding module (or trained word embedding layer).

[0117] S602: Generate a second combined feature based on the low-frequency signal feature of the second background image and the high-frequency signal feature of the random noise image.

[0118] The present application will extract the low-frequency signal features of the second background image and the high-frequency signal features of the random noise image, and generate a second combined feature based on the low-frequency signal features of the second background image and the high-frequency signal features of the random noise image. A random noise image refers to a randomly generated noise image, and a random noise image can be generated by a random number, and different random numbers correspond to different random noise images, and a random number refers to any number.

[0119] Among them, the low-frequency signal features of the second background image can reflect the background environment information such as large areas of environment and large lines in the second background image. This is because the low-frequency signal features of the image refer to the slowly changing parts or larger structures and textures in the image (representing the overall structure and background information of the image). In the frequency domain analysis, the low-frequency signal features correspond to the lower frequency parts in the spectrum. The high-frequency signal features of the image are the parts of the image that change dramatically, such as edges (contours), noise, and details. Since the high-frequency components in the random noise image are mainly noise, the high-frequency signal features of the random noise image also mainly contain noise. In the frequency domain analysis, the high-frequency signal features correspond to the higher frequency parts in the spectrum.

[0120] See Figure 8The schematic diagram of the process of generating the second combined feature shown in the figure can extract the low-frequency signal of the second background image in the latent space to obtain the low-frequency signal feature of the second background image, and (optional option: in the latent space) extract the high-frequency signal of the random noise image to obtain the high-frequency signal feature of the random noise image, and combine the low-frequency signal feature of the second background image and the high-frequency signal feature of the random noise image to obtain the second combined feature. Among them, the latent space refers to a low-dimensional space used to represent data in the field of machine learning and artificial intelligence, which is usually obtained by mapping or encoding high-dimensional data. In specific implementation, the image encoding module in the initial diffusion network can be called to perform image encoding processing on the second background image to obtain the image feature of the second background image, perform Fourier transform on the image feature of the second background image to obtain the Fourier feature of the second background image, and call a low-pass filter to perform low-pass filtering processing on the Fourier feature of the second background image (for example, frequencies below 20 are classified as low-frequency signal features) to obtain the low-frequency signal features of the second background image. Perform Fourier transform on the random noise image to obtain the Fourier features of the random noise image, call a high-pass filter to perform high-pass filtering on the Fourier features of the random noise image (for example, frequencies above 20 are classified as high-frequency signal features), and obtain high-frequency signal features of the random noise image. Alternatively, call the image encoding module in the initial diffusion network to perform image encoding on the random noise image to obtain image features of the random noise image, perform Fourier transform on the image features of the random noise image to obtain the Fourier features of the random noise image, call a high-pass filter to perform high-pass filtering on the Fourier features of the random noise image to obtain high-frequency signal features of the random noise image. Splice the high-frequency signal features of the random noise image and the low-frequency signal features of the second background image, and perform inverse Fourier transform on the features obtained by the splicing process to obtain the second combined features.

[0121] Among them, the low-pass filter is a filter used to remove high-frequency signals (such as details and noise) in the image. The low-pass filter achieves the purpose of denoising and smoothing the image by retaining the low-frequency information in the image (such as smooth transition parts and overall structure) and discarding the high-frequency information. Common low-pass filters include mean filtering and Gaussian filtering. The high-pass filter is a filter used to enhance the high-frequency signals (such as noise, edges, and textures) in the image. The high-pass filter enhances the edge clarity and texture characteristics of the image by highlighting the high-frequency detail information in the image. Common high-pass filters include Sobel filters, Laplacian filters, etc. Fourier transform is a mathematical method used to convert signals or images from the time domain to the frequency domain. In the frequency domain, the characteristics of the image can be represented by information such as frequency and phase. Fourier features usually refer to the features that describe the image in the frequency domain. If the image feature is a 3D matrix (w h c, w, h represent the width and height of the latent space image, and c represents the number of channels), then a three-dimensional Fourier transform is required. The three-dimensional Fourier transform is a mathematical technique for converting signals or data in a three-dimensional space from the spatial domain to the frequency domain. The inverse Fourier transform is the inverse operation of the Fourier transform, which is used to convert the frequency domain signal back to the time domain signal.

[0122] S603: Input the second combined feature and the third text feature of the third text into the second initial model for image generation processing to obtain a predicted embedded image.

[0123] In one implementation, a trained text encoding module (or a trained word embedding layer) may be called to perform text encoding processing on the third text to obtain a third text feature of the third text. The third text feature may be used to reflect the semantics of the third text, and the third text feature includes a feature of the second preset identifier, and the feature of the second preset identifier is bound to feature information of the second target object, which is conducive to improving the stability of the synthesis result.

[0124] Since the second combined feature is obtained by combining the low-frequency signal features of the second background image and the high-frequency signal features of the random noise image, the second combined feature includes the low-frequency signal features of background environment information such as large environments and large lines in the second background image, as well as the high-frequency signal features containing noise.

[0125] In one embodiment, the second combined feature and the third text feature of the third text are input into the second initial model for image generation processing to obtain a predicted embedded image, including: performing T times of denoising operation on the second combined feature through the forward diffusion module in the initial diffusion network to obtain the third noise feature. The third text feature of the third text is input into the second bypass network to obtain a sample bypass weight parameter; specifically, each bypass module in the second bypass network will output a sample bypass weight parameter. The backward diffusion module in the initial diffusion network is updated based on the sample bypass weight parameter. The third text feature and the third noise feature are input into the updated backward diffusion module, and the updated backward diffusion module performs backward diffusion processing on the third noise feature based on the third text feature (i.e., performs T times of denoising) to obtain an auxiliary image feature (i.e., the second combined feature restored by the updated backward diffusion module), and the auxiliary image feature is subjected to image decoding processing by the image decoding module in the initial diffusion network to obtain a predicted embedded image.

[0126] Among them, the execution logic of "inputting the third text feature of the third text into the second bypass network to obtain the sample bypass weight parameter, and updating the backward diffusion module in the initial diffusion network based on the sample bypass weight parameter. And inputting the third text feature and the third noise feature into the updated backward diffusion module, and the updated backward diffusion module performs backward diffusion processing on the third noise feature based on the third text feature" is consistent with the execution logic of the aforementioned "inputting the second text feature of the second text into the initial bypass network to obtain the initial bypass weight parameter, and updating the backward diffusion module in the initial diffusion network based on the initial bypass weight parameter. And inputting the second text feature and the second noise feature into the updated backward diffusion module, and the updated backward diffusion module performs backward diffusion processing on the second noise feature based on the second text feature", which will not be repeated here.

[0127] S604: Based on the difference between the predicted embedded image and the embedded reference image, the initial diffusion network is trained to obtain a target diffusion network.

[0128] In one embodiment, the second loss parameter may be calculated based on the difference between the predicted embedded image and the embedded reference image. The larger the difference, the larger the second loss parameter, and the smaller the difference, the smaller the second loss parameter. Exemplarily, the second loss parameter may be calculated using MSE (Mean Squared Error) loss, which may be specifically expressed as the following formula:

[0129] The second loss function =

[0130] Among them, y i represents the pixel value of each pixel in the embedded reference image, Represents the pixel value of each pixel in the predicted embedded image, and n represents the number of pixels.

[0131] Then, stochastic gradient descent (SGD) or other optimization algorithms can be used to optimize (i.e. adjust) the network parameters of the initial diffusion network in the direction of reducing the second loss parameter to realize the training process of the initial diffusion network. The "in the direction of reducing the second loss parameter" mentioned here means: the optimization direction with the goal of minimizing the second loss parameter; by optimizing in this direction, the second loss parameter generated again by the second initial model after each optimization of the initial diffusion network must be smaller than the second loss parameter generated by the second initial model before the initial diffusion network is optimized.

[0132] In the actual training process of the initial diffusion network, it is necessary to continuously obtain the second training samples to train the initial diffusion network. Each time the training is performed, the network parameters of the initial diffusion network are updated (i.e., adjusted). If the second training stop condition is reached after multiple updates, it can be determined that the training process of the initial diffusion network is completed, and the initial diffusion network obtained by the last training can be determined as the target diffusion network. The second training stop condition includes but is not limited to at least one of the following conditions: the second loss parameter is less than the set loss threshold, and the number of training times reaches the second preset number of times.

[0133] In order to improve the generalization ability of the target diffusion network, in the process of training the initial diffusion network, a second training sample with multiple samples can be obtained, that is, the second target object can refer to multiple different objects. For example, some second training samples allow the initial diffusion network in the second initial model to learn to embed object 1 into any specified background image, and the second bypass network included in the second initial model is the bypass network of object 1; some second training samples allow the initial diffusion network to learn to embed object 2 into any specified background image, and the second bypass network included in the second initial model is the bypass network of object 2; some second training samples allow the initial diffusion network to learn to embed object 3 into any specified background image, and the second bypass network included in the second initial model is the bypass network of object 3. By introducing a variety of second training samples, the initial diffusion network can learn to embed different objects into different background images, so that the target diffusion network obtained still has good performance when applied to the fusion of new personalized objects and background images.

[0134] In a feasible implementation, a second training set can be obtained, the second training set includes multiple second training samples, the second training set can be divided into multiple second training batches, each second training batch can include one or more second training samples, and one second training batch can train the initial bypass network (and the text encoding module or word embedding layer) once, and the second training set can be iterated a second preset number of times in total, and one iteration means that each second training sample in the second training set is trained once.

[0135] It can be seen that the embodiment of the present application can be based on the embedded reference image, so that the second initial model can learn how to make the predicted embedded image closer to the embedded reference image during the training process, so that the trained target diffusion network can accurately embed the personalized object into the background image. At the same time, introducing diverse second training samples during the training process of the initial diffusion network can enable the trained target diffusion network to have the generalization ability to embed any personalized object into an unknown background image, which is conducive to improving the image synthesis effect.

[0136] See also Fig. 9 , Fig. 9 This is a schematic diagram of an image processing method provided in an embodiment of the present application. Figure 3 The image processing method introduces the application process of the image generation model, and the image processing method can be performed by a model application device, such as Figure 2 The image processing method mainly includes but is not limited to the following steps S901 to S903:

[0137] S901: Acquire a first foreground image, a first background image, and a first text, wherein the first foreground image includes a first target object, and the first text is used to indicate that the first target object is embedded in the first background image.

[0138] The first background image can be any image. When the first text contains the description information of the first target object, the description information of the first target object needs to be replaced with the first preset identifier. In addition to indicating that the first target object is embedded in the first background image, the first text can also indicate other image synthesis constraints, such as Represents the first target object, and the first text may be: "Replace the xx object in the first background image with ", the first text can be " added to the first background image and located on the lawn”; the first text may also indicate other image content constraints, for example, the first text may be “add Embedded into the first background image and converted to anime style", the first text can be " Embedded into the first background image, and Wear sunglasses."

[0139] In one implementation, when the first text is used to indicate that the third target object (such as the xx object) in the first background image is replaced with the first target object, it is necessary to identify the detection image area containing the third target object from the first background image, and perform a subtraction process on the detection image area in the first background image to obtain a background template image. For example, the first background image is subjected to object or person detection, a frame diagram is taken, and the pixels in the frame diagram are set to 0. For another example, a method of object or person segmentation is adopted, and pixels of a preset ratio (such as 10% of the length and width of the external matrix of the segmentation area) are expanded outward from the segmentation edge for subtraction. Subsequent operations on the first background image are replaced with operations on the background template image, for example, image coding processing is performed on the first background image to obtain the image features of the first background image, and replaced with image coding processing is performed on the background template image to obtain the image features of the first background image. This is conducive to the image generation model to accurately locate the embedded position of the first target object.

[0140] S902: Generate a first combined feature based on the low-frequency signal feature of the first background image and the high-frequency signal feature of the random noise image.

[0141] Among them, the low-frequency signal characteristics of the first background image can reflect the background environment information such as large areas of environment and large lines in the first background image, and the high-frequency signal characteristics of the random noise image mainly include noise. In one implementation, the image encoding module in the target diffusion network can be called to perform image encoding processing on the first background image to obtain the image characteristics of the first background image, perform Fourier transform on the image characteristics of the first background image to obtain the Fourier characteristics of the first background image, call a low-pass filter to perform low-pass filtering processing on the Fourier characteristics of the first background image (for example, frequencies below 20 are classified as low-frequency signal characteristics), and obtain the low-frequency signal characteristics of the first background image. Perform Fourier transform on the random noise image to obtain the Fourier characteristics of the random noise image, call a high-pass filter to perform high-pass filtering processing on the Fourier characteristics of the random noise image (for example, frequencies above 20 are classified as high-frequency signal characteristics), and obtain the high-frequency signal characteristics of the random noise image. Alternatively, the image encoding module in the target diffusion network is called to perform image encoding processing on the random noise image to obtain image features of the random noise image, the image features of the random noise image are Fourier transformed to obtain Fourier features of the random noise image, and a high-pass filter is called to perform high-pass filtering processing on the Fourier features of the random noise image to obtain high-frequency signal features of the random noise image. The high-frequency signal features of the random noise image and the low-frequency signal features of the first background image are spliced, and the features obtained by the splicing processing are inverse Fourier transformed to obtain the first combined features.

[0142] S903. Input the first combined feature and the first text feature of the first text into an image generation model for image generation processing to obtain a composite image; wherein the image generation model includes a first bypass network constructed based on the first foreground image, and the first bypass network contains feature information of the first target object.

[0143] The image generation model includes the first bypass network and the target diffusion network. The construction process of the target diffusion network can be found in Figure 6 In one embodiment, it can be determined whether the first bypass network exists based on the description information of the first target object. When the first bypass network does not exist, the first bypass network is constructed based on the first foreground image. The construction process of the first bypass network can be seen in Figure 3 The corresponding embodiments will not be described in detail here.

[0144] In one embodiment, the first combination feature and the first text feature of the first text are input into the image generation model for image generation processing to obtain a synthetic image, including: calling the forward diffusion module in the target diffusion network to perform forward diffusion processing on the first combination feature to obtain a first noise feature. For example, the first noise feature is obtained by performing T noise addition operations on the first combination feature through the forward diffusion module in the target diffusion network. The first bypass network and the backward diffusion module in the target diffusion network are called to perform backward diffusion processing on the first noise feature based on the first text feature of the first text to obtain a reference image feature. In specific implementation, the first text feature of the first text is input into the first bypass network to obtain a bypass weight parameter (used to reflect the feature information of the first target object); specifically, each bypass module in the first bypass network outputs a bypass weight parameter. The backward diffusion module in the target diffusion network is updated based on the bypass weight parameter; the first text feature and the first noise feature are input into the updated backward diffusion module, and the updated backward diffusion module performs backward diffusion processing on the first noise feature based on the first text feature to obtain a reference image feature. Among them, the execution logic of "inputting the first text feature of the first text into the first bypass network to obtain the bypass weight parameter; updating the back diffusion module in the target diffusion network based on the bypass weight parameter; inputting the first text feature and the first noise feature into the updated back diffusion module, and the updated back diffusion module performs back diffusion processing on the first noise feature based on the first text feature" is consistent with the execution logic of "inputting the second text feature of the second text into the initial bypass network to obtain the initial bypass weight parameter, and updating the back diffusion module in the initial diffusion network based on the initial bypass weight parameter. And inputting the second text feature and the second noise feature into the updated back diffusion module, and the updated back diffusion module performs back diffusion processing on the second noise feature based on the second text feature", which will not be repeated here. The image decoding module in the target diffusion network can be called to perform image decoding processing on the reference image feature to obtain a synthetic image.

[0145] In summary, see Fig.10a , Fig.10a This is a schematic diagram of an image processing method provided in an embodiment of the present application. Figure 4 In the embodiment of the present application, the input (i.e., combined features) of the image generation model will be generated through the low-frequency signal features of the specified background image and the high-frequency signal features of the random noise image. The image generation model will generate a synthetic image that is consistent with the personalized object in terms of lighting, color, and large background under the control of the bypass network of the text and personalized object, avoiding the problem of poor generation when the personalized object is greatly different from the background image. Please refer to Fig.10b , Fig.10b is a schematic diagram of the effect of a synthetic image provided by an embodiment of the present application, which can be Fig.10bThe left image is taken as the first background image, the person in the left image is taken as the first target object, and the first text is: "Embed the first target object into the first background image, and transform the first target object into the next running action". By inputting the first text and the first background image into the image generation model, a synthetic image (i.e. Fig.10b This method can achieve continuous image generation.

[0146] It can be seen that, in the embodiments of the present application, on the one hand, based on the low-frequency signal features, the image generation model ensures that the background environment information such as large environments and large lines in the first background image can be retained in the composite image, while the high-frequency signal features that determine more picture details in the composite image can be adjusted with the help of the control signal (i.e., the first text features of the first text) to support the limited background adaptive change of the first background image; on the other hand, the image generation model can support the joint fine-tuning of the first target object based on the first bypass network, such as more accurately adjusting the high-frequency signal features that determine the picture details with the help of the feature information of the first target object, so that the background area of ​​the composite image can maintain a consistent effect with the first target object, and finally obtain a composite image with coordinated effects.

[0147] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0148] Fig.11 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application; the image processing device can be used to perform Figure 3 , Figure 6 or Fig. 9 Some or all of the steps in the method embodiment shown. Fig.11 The image processing device includes the following units: an acquisition unit 1101 and a processing unit 1102.

[0149] An acquisition unit 1101 is used to acquire a first foreground image, a first background image and a first text, wherein the first foreground image includes a first target object, and the first text is used to indicate that the first target object is embedded in the first background image;

[0150] A processing unit 1102 is used to generate a first combined feature based on the low-frequency signal feature of the first background image and the high-frequency signal feature of the random noise image;

[0151] The processing unit 1102 is also used to input the first combined feature and the first text feature of the first text into the image generation model for image generation processing to obtain a composite image; wherein the image generation model includes a first bypass network constructed based on the first foreground image, and the first bypass network contains feature information of the first target object.

[0152] In one embodiment, the image generation model includes a target diffusion network and a first bypass network; the processing unit 1102 is used to input the first combined feature and the first text feature of the first text into the image generation model to perform image generation processing, and when the synthetic image is obtained, the following steps are specifically performed:

[0153] Calling the forward diffusion module in the target diffusion network to perform forward diffusion processing on the first combined feature to obtain a first noise feature;

[0154] Calling the back diffusion modules in the first bypass network and the target diffusion network to perform back diffusion processing on the first noise feature based on the first text feature of the first text to obtain a reference image feature;

[0155] The reference image features are subjected to image decoding processing to obtain a synthetic image.

[0156] In one embodiment, the processing unit 1102 is used to call the back diffusion module in the first bypass network and the target diffusion network, and perform back diffusion processing on the first noise feature based on the first text feature of the first text to obtain the reference image feature, specifically performing the following steps:

[0157] Inputting a first text feature of a first text into a first bypass network to obtain a bypass weight parameter;

[0158] Updating the backward diffusion module in the target diffusion network based on the bypass weight parameters;

[0159] The first text feature and the first noise feature are input into the updated back diffusion module, and the updated back diffusion module performs back diffusion processing on the first noise feature based on the first text feature to obtain a reference image feature.

[0160] In one embodiment, the acquisition unit 1101 is further configured to perform the following steps:

[0161] Acquire a first training sample and a first initial model, wherein the first training sample includes a first foreground image and a second text, wherein the second text includes description information of a first target object; and the first initial model includes an initial bypass network and an initial diffusion network;

[0162] The processing unit 1102 is further configured to perform the following steps:

[0163] Inputting a first image feature of a first foreground image and a second text feature of a second text into a first initial model, and predicting a first noise added to the first image feature by the first initial model;

[0164] Based on the difference between the first noise added to the predicted first image feature and the second noise actually added to the first image feature, the initial bypass network is trained to obtain the first bypass network.

[0165] In one embodiment, the processing unit 1102 is used to input the first image feature of the first foreground image and the second text feature of the second text into the first initial model, and when the first initial model predicts the first noise added to the first image feature, specifically perform the following steps:

[0166] Calling the forward diffusion module in the initial diffusion network to perform forward diffusion processing on the first image feature to obtain a second noise feature, where the second noise feature is obtained by adding the second noise to the first image feature;

[0167] Calling the back diffusion module in the initial bypass network and the initial diffusion network to perform back diffusion processing on the second noise feature based on the second text feature of the second text;

[0168] A first noise added to a first image feature predicted during a back diffusion process is obtained.

[0169] In one embodiment, the acquisition unit 1101 is further configured to perform the following steps:

[0170] Acquire a second training sample and a second initial model, wherein the second training sample includes a second background image, a third text, and an embedded reference image, wherein the third text is used to indicate that a second target object is embedded in the second background image; the second initial model includes an initial diffusion network and a second bypass network constructed based on a second foreground image, wherein the second foreground image includes a second target object;

[0171] The processing unit 1102 is further configured to perform the following steps:

[0172] Inputting the second combined feature and the third text feature of the third text into the second initial model to perform image generation processing to obtain a predicted embedded image;

[0173] Based on the difference between the predicted embedded image and the embedded reference image, the initial diffusion network is trained to obtain the target diffusion network.

[0174] In one embodiment, when the processing unit 1102 is used to generate the first combined feature based on the low-frequency signal feature of the first background image and the high-frequency signal feature of the random noise image, the processing unit 1102 specifically performs the following steps:

[0175] Calling a low-pass filter to perform low-pass filtering on the Fourier features of the first background image to obtain low-frequency signal features of the first background image;

[0176] Calling a high-pass filter to perform high-pass filtering on the Fourier characteristics of the random noise image to obtain the high-frequency signal characteristics of the random noise image;

[0177] Performing splicing processing on the low-frequency signal features of the first background image and the high-frequency signal features of the random noise image to obtain spliced ​​signal features;

[0178] Perform inverse Fourier transform on the concatenated signal features to obtain the first combined features.

[0179] In one embodiment, the first text is specifically used to indicate that the third target object in the first background image is replaced with the first target object; the processing unit 1102 is further used to perform the following steps:

[0180] identifying a detection image region containing a third target object from the first background image;

[0181] Performing a cutout process on the detection image area in the first background image to obtain a background template image;

[0182] Perform Fourier transform on the background template image to obtain Fourier features of the first background image.

[0183] According to one embodiment of the present application, Fig.11 The various units in the image processing device shown can be individually or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the image processing device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, the following can be executed by running on a general computing device such as a computer including processing elements and storage elements such as a central processing unit 1102 (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM). Figure 3 , Figure 6 or Fig. 9 A computer program (including program code) for each step involved in the corresponding method shown in FIG. Fig.11 The image processing apparatus shown in the figure can be used to implement the image processing method of the embodiment of the present application. The computer program can be recorded on a computer-readable recording medium, for example, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0184] The technical solution provided by the embodiment of the present application can obtain a first foreground image, a first background image and a first text, wherein the first foreground image contains a first target object, and the first text is used to indicate that the first target object is embedded in the first background image. A first combined feature is generated based on the low-frequency signal features of the first background image and the high-frequency signal features of the random noise image; the low-frequency signal features of the first background image contain background environment information such as large areas of environment and large sections of lines in the first background image, and the high-frequency signal features of the random noise image mainly contain noise. The first combined feature and the first text feature of the first text are input into the image generation model for image generation processing to obtain a composite image. On the one hand, since the low-frequency signal features will exist in the information flow of the image generation model (referring to the information transmission and aggregation process within the model), it can be ensured that the background environment information such as large areas of environment and large lines in the first background image can be retained in the synthetic image. At the same time, the high-frequency signal features that determine more picture details in the synthetic image can be adjusted with the help of a control signal (i.e., the first text feature of the first text) to support the limited background adaptive change of the first background image; on the other hand, the image generation model includes a first bypass network constructed based on the first foreground image, and the first bypass network contains feature information of the first target object. Therefore, the image generation model can support joint fine-tuning of the first target object, such as using the feature information of the first target object to more accurately adjust the high-frequency signal features that determine the picture details, so that the background area of ​​the synthetic image can maintain a consistent effect with the first target object, and finally obtain a synthetic image with coordinated effects.

[0185] Fig.12 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.12 , the computer device includes a processor 1201, a communication interface 1202 and a computer-readable storage medium 1203. The processor 1201, the communication interface 1202 and the computer-readable storage medium 1203 can be connected via a bus or other means. The communication interface 1202 is used to receive and send data. The computer-readable storage medium 1203 can be stored in the memory of the computer device, the computer-readable storage medium 1203 is used to store a computer program, the computer program includes program instructions, and the processor 1201 is used to execute the program instructions stored in the computer-readable storage medium 1203. The processor 1201 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function.

[0186] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the processing system of the computer device. In addition, one or more instructions suitable for being loaded and executed by the processor 1201 are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0187] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor 1201 loads and executes the one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above-mentioned image processing method embodiment; in a specific implementation, the one or more instructions in the computer-readable storage medium are loaded by the processor 1201 and the following steps are executed:

[0188] Acquire a first foreground image, a first background image, and a first text, wherein the first foreground image includes a first target object, and the first text is used to indicate that the first target object is embedded in the first background image;

[0189] Generate a first combined feature based on the low-frequency signal feature of the first background image and the high-frequency signal feature of the random noise image;

[0190] The first combined feature and the first text feature of the first text are input into an image generation model for image generation processing to obtain a composite image; wherein the image generation model includes a first bypass network constructed based on the first foreground image, and the first bypass network contains feature information of the first target object.

[0191] In one embodiment, the image generation model includes a target diffusion network and a first bypass network; the processor 1201 is used to input the first combined feature and the first text feature of the first text into the image generation model to perform image generation processing, and when the synthesized image is obtained, the following steps are specifically performed:

[0192] Calling the forward diffusion module in the target diffusion network to perform forward diffusion processing on the first combined feature to obtain a first noise feature;

[0193] Calling the back diffusion modules in the first bypass network and the target diffusion network to perform back diffusion processing on the first noise feature based on the first text feature of the first text to obtain a reference image feature;

[0194] The reference image features are subjected to image decoding processing to obtain a synthetic image.

[0195] In one embodiment, the processor 1201 is used to call the back diffusion module in the first bypass network and the target diffusion network, and perform back diffusion processing on the first noise feature based on the first text feature of the first text to obtain the reference image feature, and specifically perform the following steps:

[0196] Inputting a first text feature of a first text into a first bypass network to obtain a bypass weight parameter;

[0197] Updating the backward diffusion module in the target diffusion network based on the bypass weight parameters;

[0198] The first text feature and the first noise feature are input into the updated back diffusion module, and the updated back diffusion module performs back diffusion processing on the first noise feature based on the first text feature to obtain a reference image feature.

[0199] In one embodiment, the processor 1201 is further configured to perform the following steps:

[0200] Acquire a first training sample and a first initial model, wherein the first training sample includes a first foreground image and a second text, wherein the second text includes description information of a first target object; and the first initial model includes an initial bypass network and an initial diffusion network;

[0201] Inputting a first image feature of a first foreground image and a second text feature of a second text into a first initial model, and predicting a first noise added to the first image feature by the first initial model;

[0202] Based on the difference between the first noise added to the predicted first image feature and the second noise actually added to the first image feature, the initial bypass network is trained to obtain the first bypass network.

[0203] In one embodiment, the processor 1201 is used to input the first image feature of the first foreground image and the second text feature of the second text into the first initial model, and when the first initial model predicts the first noise added to the first image feature, specifically perform the following steps:

[0204] Calling the forward diffusion module in the initial diffusion network to perform forward diffusion processing on the first image feature to obtain a second noise feature, where the second noise feature is obtained by adding the second noise to the first image feature;

[0205] Calling the back diffusion module in the initial bypass network and the initial diffusion network to perform back diffusion processing on the second noise feature based on the second text feature of the second text;

[0206] A first noise added to a first image feature predicted during a back diffusion process is obtained.

[0207] In one embodiment, the processor 1201 is further configured to perform the following steps:

[0208] Acquire a second training sample and a second initial model, wherein the second training sample includes a second background image, a third text, and an embedded reference image, wherein the third text is used to indicate that a second target object is embedded in the second background image; the second initial model includes an initial diffusion network and a second bypass network constructed based on a second foreground image, wherein the second foreground image includes a second target object;

[0209] Inputting the second combined feature and the third text feature of the third text into the second initial model to perform image generation processing to obtain a predicted embedded image;

[0210] Based on the difference between the predicted embedded image and the embedded reference image, the initial diffusion network is trained to obtain the target diffusion network.

[0211] In one embodiment, when the processor 1201 is used to generate the first combined feature based on the low-frequency signal feature of the first background image and the high-frequency signal feature of the random noise image, the processor 1201 specifically performs the following steps:

[0212] Calling a low-pass filter to perform low-pass filtering on the Fourier features of the first background image to obtain low-frequency signal features of the first background image;

[0213] Calling a high-pass filter to perform high-pass filtering on the Fourier characteristics of the random noise image to obtain the high-frequency signal characteristics of the random noise image;

[0214] Performing splicing processing on the low-frequency signal features of the first background image and the high-frequency signal features of the random noise image to obtain spliced ​​signal features;

[0215] Perform inverse Fourier transform on the concatenated signal features to obtain the first combined features.

[0216] In one embodiment, the first text is specifically used to instruct to replace the third target object in the first background image with the first target object; the processor 1201 is further used to perform the following steps:

[0217] identifying a detection image region containing a third target object from the first background image;

[0218] Performing a cutout process on the detection image area in the first background image to obtain a background template image;

[0219] Perform Fourier transform on the background template image to obtain Fourier features of the first background image.

[0220] Based on the same inventive concept, the principles and beneficial effects of solving the problems provided by the computer device in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems provided by the image processing method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0221] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0222] The embodiment of the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned image processing method.

[0223] A person skilled in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0224] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or an image processing device such as a server, a data center, etc. that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0225] In this application, elements expressed in the singular are intended to mean "one or more" rather than "one and only one", unless otherwise specified. In this application, "at least one" is intended to mean "one or more", and "more than one" is intended to mean "two or more", unless otherwise specified.

[0226] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technical object familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first foreground image, a first background image, and a first text, wherein the first foreground image includes a first target object, and the first text is used to indicate that the first target object is embedded in the first background image; generating a first combined feature based on the low-frequency signal feature of the first background image and the high-frequency signal feature of the random noise image; Inputting the first combined feature and the first text feature of the first text into an image generation model; wherein the image generation model includes a target diffusion network and a first bypass network constructed based on the first foreground image, and the first bypass network contains feature information of the first target object; Calling a forward diffusion module in the target diffusion network to perform forward diffusion processing on the first combined feature to obtain a first noise feature; Calling the back diffusion modules in the first bypass network and the target diffusion network to perform back diffusion processing on the first noise feature based on the first text feature of the first text to obtain a reference image feature; Image decoding processing is performed on the reference image features to obtain a synthesized image.

2. The method according to claim 1, characterized in that The calling of the first bypass network and the back diffusion module in the target diffusion network to perform back diffusion processing on the first noise feature based on the first text feature of the first text to obtain a reference image feature includes: Inputting the first text feature of the first text into the first bypass network to obtain a bypass weight parameter; Updating a backward diffusion module in the target diffusion network based on the bypass weight parameter; The first text feature and the first noise feature are input into an updated back diffusion module, and the updated back diffusion module performs back diffusion processing on the first noise feature based on the first text feature to obtain a reference image feature.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Acquire a first training sample and a first initial model, wherein the first training sample includes the first foreground image and a second text, wherein the second text includes description information of the first target object; and the first initial model includes an initial bypass network and an initial diffusion network; Inputting a first image feature of the first foreground image and a second text feature of the second text into the first initial model, and predicting a first noise added to the first image feature by the first initial model; The initial bypass network is trained based on a difference between a first noise predicted to be added to the first image feature and a second noise actually added to the first image feature to obtain the first bypass network.

4. The method according to claim 3, characterized in that The step of inputting the first image feature of the first foreground image and the second text feature of the second text into the first initial model, and predicting the first noise added to the first image feature by the first initial model, comprises: calling a forward diffusion module in the initial diffusion network to perform forward diffusion processing on the first image feature to obtain a second noise feature, where the second noise feature is obtained by adding a second noise to the first image feature; Calling the back diffusion modules in the initial bypass network and the initial diffusion network to perform back diffusion processing on the second noise feature based on the second text feature of the second text; A first noise added to the first image feature predicted during the backward diffusion process is obtained.

5. The method according to claim 1, characterized in that The method further comprises: Acquire a second training sample and a second initial model, wherein the second training sample includes a second background image, a third text, and an embedded reference image, wherein the third text is used to indicate that a second target object is embedded in the second background image; the second initial model includes an initial diffusion network and a second bypass network constructed based on a second foreground image, wherein the second foreground image includes the second target object; generating a second combined feature based on the low-frequency signal feature of the second background image and the high-frequency signal feature of the random noise image; Inputting the second combined feature and the third text feature of the third text into the second initial model to perform image generation processing to obtain a predicted embedded image; Based on the difference between the predicted embedded image and the embedded reference image, the initial diffusion network is trained to obtain the target diffusion network.

6. The method according to claim 1, characterized in that The generating a first combined feature based on the low-frequency signal feature of the first background image and the high-frequency signal feature of the random noise image comprises: Calling a low-pass filter to perform low-pass filtering on the Fourier features of the first background image to obtain low-frequency signal features of the first background image; Calling a high-pass filter to perform high-pass filtering on the Fourier characteristics of the random noise image to obtain high-frequency signal characteristics of the random noise image; Performing splicing processing on the low-frequency signal features of the first background image and the high-frequency signal features of the random noise image to obtain spliced ​​signal features; Perform an inverse Fourier transform on the spliced ​​signal feature to obtain a first combined feature.

7. The method according to claim 6, characterized in that The first text is specifically used to indicate that the third target object in the first background image is replaced by the first target object; the method further includes: Identifying a detection image region containing the third target object from the first background image; Performing a subtraction process on the detection image area in the first background image to obtain a background template image; Performing Fourier transform on the background template image to obtain Fourier features of the first background image.

8. An image processing device, characterized in that: The device comprises: An acquisition unit, configured to acquire a first foreground image, a first background image, and a first text, wherein the first foreground image includes a first target object, and the first text is used to indicate that the first target object is embedded in the first background image; A processing unit, configured to generate a first combined feature based on the low-frequency signal feature of the first background image and the high-frequency signal feature of the random noise image; The processing unit is further used to input the first combined feature and the first text feature of the first text into an image generation model for image generation processing to obtain a synthetic image; wherein the image generation model includes a target diffusion network and a first bypass network constructed based on the first foreground image, and the first bypass network contains feature information of the first target object; Calling a forward diffusion module in the target diffusion network to perform forward diffusion processing on the first combined feature to obtain a first noise feature; Calling the back diffusion modules in the first bypass network and the target diffusion network to perform back diffusion processing on the first noise feature based on the first text feature of the first text to obtain a reference image feature; Image decoding processing is performed on the reference image features to obtain a synthesized image.

9. A computer device, characterized in that: include: a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the image processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the image processing method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.

Images