Image processing method and device, electronic equipment and medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: receiving first input of a user; generating image description information in response to the first input; generating a first image corresponding to the image description information based on the image description information; generating a second image corresponding to a target part area based on target part description information corresponding to the target part area in the image description information under the condition of determining that the first image comprises the target part area; and replacing the target part area in the first image with the second image to obtain a third image.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an image processing method, device, electronic device and medium. Background Art

[0002] With the rise of artificial intelligence generated content (AIGC) technology, functions such as text-to-image generation, AI passerby removal, image expansion, and image editing have entered the daily lives of consumers. The technologies behind these functions are text-to-image algorithms.

[0003] When the Wenshengtu algorithm currently available on the market generates character images, some parts of the character images are prone to deformity, which results in poor effects of character images generated by the existing Wenshengtu algorithm. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and medium to generate a target part image with better quality, thereby solving the problem of target part deformity in a person image generated based on a Vincent map algorithm.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0006] receiving a first input from a user;

[0007] In response to the first input, generating image description information;

[0008] Based on the image description information, generating a first image corresponding to the image description information;

[0009] In a case where it is determined that the first image includes a target part, generating a second image corresponding to the target part based on the target part description information corresponding to the target part in the image description information;

[0010] The target part in the first image is replaced by the second image to obtain a third image.

[0011] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0012] A first receiving module, used to receive a first input from a user;

[0013] A first generating module, configured to generate image description information in response to the first input;

[0014] A second generating module, configured to generate a first image corresponding to the image description information based on the image description information;

[0015] a third generating module, configured to generate a second image corresponding to the target part based on the target part description information corresponding to the target part in the image description information when it is determined that the first image includes the target part;

[0016] The fourth generating module is used to replace the target part in the first image with the second image to obtain a third image.

[0017] In a third aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0020] In an embodiment of the present application, image description information is generated in response to a first input from a user, and then based on the image description information, a first image corresponding to the image description information can be generated. When it is determined that the first image includes a target part, a second image corresponding to the target part can be separately generated based on the target part description information corresponding to the target part in the image description information. The target part in the second image generated in this way has a better effect and will not have deformity problems. Then, the target part in the first image is replaced with the second image to obtain a third image, thereby solving the problem of deformity of the target part in the character image generated based on the Vincent graph algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart of an image processing method provided by some embodiments of the present application;

[0022] Figure 2 is a schematic diagram of an interface corresponding to AIGC provided in some embodiments of the present application;

[0023] Figure 3 is a schematic diagram of an interface corresponding to AIGC provided in some embodiments of the present application;

[0024] Figure 4is a schematic diagram of an interface corresponding to AIGC provided in some embodiments of the present application;

[0025] Figure 5 is a schematic diagram of an algorithm framework for generating a first image and a second image provided by some embodiments of the present application;

[0026] Figure 6 is a flow chart of the internal processing process of the to-be-trained text encoder provided by some embodiments of the present application;

[0027] Figure 7 is a flow chart of the internal processing process of the to-be-trained text encoder provided by some embodiments of the present application;

[0028] Figure 8 is a flow chart of the internal processing process of the image encoder to be trained provided by some embodiments of the present application;

[0029] Fig. 9 is a flow chart of the internal processing process of the image encoder to be trained provided by some embodiments of the present application;

[0030] Fig.10 is a flow chart of the internal processing process of the diffusion model to be trained provided by some embodiments of the present application;

[0031] Fig.11 is a flow chart of the internal processing process of the target part reconstruction model to be trained provided by some embodiments of the present application;

[0032] Fig.12 is a flowchart of an image processing method provided by some embodiments of the present application;

[0033] Fig.13 is a schematic diagram of the structure of an information processing device shown in some embodiments of the present application;

[0034] Fig.14 is a schematic diagram of the structure of an electronic device shown in some embodiments of the present application;

[0035] Fig.15 It is a schematic diagram of the hardware structure of an electronic device shown in some embodiments of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0037] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or N. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0038] The technical solution of the embodiment of the present application can be applied to a scene in which a target part in a character image generated based on a Vincent image algorithm is deformed, such as a scene in which a hand is deformed in a character image generated based on a Vincent image algorithm.

[0039] Before introducing the solution of the embodiment of the present application, the professional terms involved in the embodiment of the present application are first introduced:

[0040] Tokenizer: It is a tool commonly used in natural language processing to break down text into words, phrases, or symbols, and is used to convert text into a digital form that the model can understand.

[0041] Variational Autoencoder (VAE): It is a generative model specifically designed to capture the underlying probability distribution of a given dataset and generate new samples. It contains the architecture of the encoder-decoder structure. The encoder converts the input data into a latent form, and the decoder aims to reconstruct the original data based on this latent representation.

[0042] Diffusion Transformer (DiT): is a diffusion model combined with the Transformer architecture for image and video generation tasks. It can efficiently capture dependencies between data and generate high-quality results.

[0043] Self-attention operator SelfAttention: In the attention mechanism, each query pays attention to all key-value pairs and generates an attention output. If the query q, key k and value v all come from the same set of inputs, then this attention is called self-attention.

[0044] The image processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0045] Figure 1It is a flow chart of an image processing method provided in an embodiment of the present application. The executor of the image processing method may be an electronic device, which may be but is not limited to a personal computer (PC), a smart phone, a tablet computer or a personal digital assistant (PDA).

[0046] like Figure 1 As shown, the image processing method provided in the embodiment of the present application may include steps 110 to 150.

[0047] Step 110: Receive a first input from a user.

[0048] Among them, the first input is the input in the interface corresponding to AIGC, and the above-mentioned first input is used to generate image description information. Exemplarily, the above-mentioned first input includes but is not limited to: the touch input of the user through a touch device such as a finger or a stylus in the interface corresponding to AIGC, or a voice command input by the user, or a specific gesture input by the user, or other feasible input, which can be determined according to actual use requirements, and the embodiment of the present invention is not limited. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input. For example, the above-mentioned first input can be: the user fills in the input in the interface corresponding to AIGC.

[0049] It should be noted that the interface corresponding to AIGC may be an interface developed based on AIGC technology for users to input image description information.

[0050] Step 120: Generate image description information in response to the first input.

[0051] The image description information may be information used to describe an image that a user wants to generate.

[0052] In one example, reference Figure 2 , the user wants to generate a picture of a girl wearing a T-shirt and making a V-sign with her right hand, then the user can Figure 2 In the interface corresponding to the AIGC shown, input “Please help me generate a picture of a girl wearing a T-shirt and making a V-shaped gesture with her right hand”, then “Please help me generate a picture of a girl wearing a T-shirt and making a V-shaped gesture with her right hand” is the image description information.

[0053] Step 130: Based on the image description information, generate a first image corresponding to the image description information.

[0054] The first image may be an image generated corresponding to the image description information.

[0055] In some embodiments of the present application, according to the image description information, a first image corresponding to the image description information may be generated based on a Vincent graph algorithm.

[0056] Continuing with the above example, after the user enters "Please help me generate a picture of a girl wearing a T-shirt and making a V-shaped gesture with her right hand" in the corresponding interface of AIGC, a picture of a girl wearing a T-shirt and making a V-shaped gesture with her right hand can be generated based on AIGC technology, such as Figure 3 shown.

[0057] Step 140: When it is determined that the first image includes the target part, a second image corresponding to the target part is generated based on the target part description information corresponding to the target part in the image description information.

[0058] The target part may be a part with a deformity in the first image, such as a hand, an eye, an ear, etc. Figure 3 In the generated image, the fingers of the person are stuck together, which does not look very nice.

[0059] The target part description information may be description information corresponding to the target part in the image description information.

[0060] The second image may be an image corresponding to the target part, which is generated based on the target part description information, that is, the second image only includes the target part.

[0061] In some embodiments of the present application, when it is determined that the first image includes a target part, a second image corresponding to the target part can be generated based on the target part description information corresponding to the target part in the image description information. Specifically, when it is determined that the first image includes the target part and an instruction from the user to repair the target part is received, a second image corresponding to the target part can be generated based on the target part description information corresponding to the target part in the image description information.

[0062] In some embodiments, in order to improve the flexibility of the target site repair, step 140 may specifically include:

[0063] In the case where it is determined that the first image includes the target part, receiving a second input from the user;

[0064] In response to the second input, generating a target site repair instruction;

[0065] Based on the target part repair instruction, a second image corresponding to the target part is generated according to the target part description information corresponding to the target part in the image description information.

[0066] Among them, the second input is the input in the interface corresponding to AIGC, and the above second input is used to generate a target part repair instruction.

[0067] The target part repair instruction may be an instruction to repair the target part.

[0068] In some embodiments of the present application, a target part repair instruction can be generated in response to a second input from the user, and then based on the target part repair instruction, a second image corresponding to the target part is generated according to the target part description information corresponding to the target part in the image description information.

[0069] Continuing with the above example, when generating Figure 3 After seeing the picture, the user wants to Figure 3 The hands of the character images generated in the above example can be repaired as follows: Figure 3 As shown, by inputting "hand repair" in the interface corresponding to AIGC, a hand repair instruction can be generated. Based on the hand repair instruction, an image of the hand is generated according to the hand description information corresponding to the hand in the image description information.

[0070] In an embodiment of the present application, a target part repair instruction is generated in response to a second input from a user, and then based on the target part repair instruction, a second image corresponding to the target part is generated according to the target part description information corresponding to the target part in the image description information. In this way, when repairing the target part, it can be executed according to the second input from the user. In this way, the target part can be repaired according to user needs, thereby improving the flexibility of target part repair.

[0071] Step 150: Replace the target part in the first image with the second image to obtain a third image.

[0072] The third image may be an image obtained by replacing the target part in the first image with the second image.

[0073] In some embodiments of the present application, the target part in the first image can be replaced by the second image, so that a person image with a non-deformed target part can be generated, such as Figure 4 As shown, in Figure 4 In the generated image of a person, the hands have no deformities.

[0074] In some embodiments of the present application, step 130 may specifically include:

[0075] Extract text features in the image description information based on the text encoder to obtain a text feature vector corresponding to the image description information;

[0076] Extract features of random images based on an image encoder and generate an image feature matrix corresponding to the random images;

[0077] Add random noise to the image feature matrix to obtain the image feature matrix after adding noise;

[0078] Based on the diffusion model, the text feature vector is used to denoise the image feature matrix after adding noise, and the image feature matrix after removing noise is obtained;

[0079] The image feature matrix after noise removal is decoded based on an image decoder to obtain a first image.

[0080] The text encoder may be a device for extracting text features from image description information.

[0081] The text feature vector may be a feature vector corresponding to the image description information obtained after the text features in the image description information are extracted by a text encoder.

[0082] The image encoder can be a device for extracting features from a random image to obtain an image feature matrix corresponding to the random image. The specific function of the image encoder is to compress the image from the pixel space to the feature space.

[0083] The random image may be an image selected at random, and the random image may be a completely white image.

[0084] The image feature matrix may be a feature matrix obtained by extracting features of a random image using an image encoder. It should be noted that all elements in the image feature matrix are 0.

[0085] The random noise may be randomly selected noise, such as Gaussian noise.

[0086] The diffusion model may be a model that uses the text feature vector to perform denoising on the image feature matrix after adding noise to obtain the image feature matrix after removing the noise. Specifically, the diffusion model may be a model for reconstructing the entire character image.

[0087] The image decoder may be a device for decoding the image feature matrix after noise removal to obtain a target image. The specific function of the image decoder is to decode the image from the feature space to the pixel space.

[0088] In some embodiments of the present application, text features in image description information can be extracted based on a text encoder to obtain a text feature vector corresponding to the image description information; feature extraction can be performed on a random image based on an image encoder to generate an image feature matrix corresponding to the random image; random noise can be added to the image feature matrix to obtain an image feature matrix after adding noise; denoising can be performed on the image feature matrix after adding noise using the text feature vector based on a diffusion model to obtain an image feature matrix after removing noise; and the image feature matrix after removing noise can be decoded based on an image decoder to obtain a first image.

[0089] Continuing with the above example, let's take the target part as the hand and the random noise as Gaussian noise. Figure 5 Schematic diagram of the algorithm framework for generating the first image and the second image. Figure 5 In , the image description information "Please help me generate a picture of a girl wearing a T-shirt and making a V-shaped gesture with her right hand" is input into the text encoder huge, and then the text features in the image description information "Please help me generate a picture of a girl wearing a T-shirt and making a V-shaped gesture with her right hand" are extracted based on the text encoder huge to obtain the text feature vector TextEmbedding1 corresponding to the image description information. Based on the image encoder VAE-Encoder, feature extraction is performed on the random image to generate the image feature matrix Latent corresponding to the random image. Gaussian noise is added to the image feature matrix Latent to obtain the image feature matrix Noise Latent after adding noise. The text feature vector TextEmbedding1 and the image feature matrix Noise Latent after adding noise are input into the diffusion model. Based on the diffusion model, the text feature vector TextEmbedding1 is used to denoise the image feature matrix Noise Latent after adding noise to obtain the image feature matrix after removing noise. Based on the image decoder VAE-Decoder, the image feature matrix after removing noise is decoded to obtain the character image 1, where the character image 1 is the first image. In Figure 5 In the figure, the hand in image 1 is deformed.

[0090] In some embodiments of the present application, generating a second image corresponding to the target part based on the target part description information corresponding to the target part in the image description information in step 140 may specifically include:

[0091] Extracting text features in the target part description information based on a text encoder to obtain a text feature vector corresponding to the target part description information;

[0092] Extract features of random images based on an image encoder and generate an image feature matrix corresponding to the random images;

[0093] Add random noise to the image feature matrix to obtain the image feature matrix after adding noise;

[0094] Based on the target part reconstruction model, the text feature vector is used to perform denoising on the image feature matrix after adding noise, and the image feature matrix after removing noise is obtained;

[0095] The image feature matrix after the noise is removed is decoded based on the image decoder to obtain a second image.

[0096] The above-mentioned text encoder may be a device for extracting text features from target part description information.

[0097] The text feature vector may be a feature vector corresponding to the target description information obtained after the text features in the target description information are extracted by a text encoder.

[0098] The image encoder can be a device for extracting features from a random image to obtain an image feature matrix corresponding to the random image. The specific function of the image encoder is to compress the image from the pixel space to the feature space.

[0099] The random image may be an image selected at random, and the random image may be a completely white image.

[0100] The image feature matrix may be a feature matrix obtained by extracting features of a random image using an image encoder. It should be noted that all elements in the image feature matrix are 0.

[0101] The target part reconstruction model may be a model of the image feature matrix after noise addition by denoising the image feature matrix using the text feature vector. Specifically, the target part reconstruction model may be a model for reconstructing the target part.

[0102] The image decoder may be a device for decoding the image feature matrix after noise removal to obtain a target image. The specific function of the image decoder is to decode the image from the feature space to the pixel space.

[0103] In some embodiments of the present application, text features in the target part description information can be extracted based on a text encoder to obtain a text feature vector corresponding to the target part description information; features of a random image can be extracted based on an image encoder to generate an image feature matrix corresponding to the random image; random noise is added to the image feature matrix to obtain an image feature matrix after adding noise; the image feature matrix after adding noise is denoised using the text feature vector based on a hand reconstruction model to obtain an image feature matrix after removing noise; the image feature matrix after removing noise is decoded based on an image decoder to obtain a second image.

[0104] exist Figure 5 In the figure, since the hand in the person image 1 is deformed, the hand area in the person image 1 can be replaced with the hand image 2 to obtain the person image 3 with no deformity in the hand.

[0105] It should be noted that the internal processing procedures of the text encoder huge used to generate the first image are consistent with those of the text encoder small used to generate the second image. Similarly, the internal processing procedures of the image encoder used to generate the first image are consistent with those of the image encoder used to generate the second image, and the internal processing procedures of the image decoder used to generate the first image are consistent with those of the image decoder used to generate the second image. It’s just that the transformation layers in the text encoder small used to generate the second image are fewer than those in the text encoder huge used to generate the first image. Specifically, the transformation layers in the text encoder small are stacked in 12 layers, while the transformation layers in the text encoder huge are stacked in 36 layers.

[0106] Correspondingly, the number of convolution kernels in the convolution layer of the image encoder that generates the first image is different from the number of convolution kernels in the convolution layer of the image encoder that generates the second image. The number of convolution kernels in the convolution layer of the image decoder that generates the first image is different from the number of convolution kernels in the convolution layer of the image decoder that generates the second image. The specific internal processing of the text encoder, image encoder and image decoder that generate the first image, and the internal processing of the text encoder, image encoder and image decoder that generate the second image will be described in detail in subsequent embodiments.

[0107] In an embodiment of the present application, a second image may be obtained through a text encoder, an image encoder, a target part reconstruction model and an image decoder, thereby improving the efficiency and accuracy of generating the second image corresponding to the target part description information.

[0108] In some embodiments of the present application, before using a text encoder, an image encoder, a diffusion model and an image decoder for generating a first image, and a text encoder, an image encoder, a target part reconstruction model and an image decoder for generating the first image, it is first necessary to train the text encoder, the image encoder, the diffusion model and the image decoder for generating the first image, and train the text encoder, the image encoder, the target part reconstruction model and the image decoder for generating the first image, so as to obtain the trained text encoder, the image encoder, the diffusion model and the image decoder for generating the first image, and the text encoder, the image encoder, the target part reconstruction model and the image decoder for generating the second image, so that the first image can be generated based on the trained text encoder, the image encoder, the diffusion model and the image decoder for generating the first image, and the second image can be generated based on the trained text encoder, the image encoder, the diffusion model and the image decoder for generating the second image.

[0109] That is, before step 110, the above method may further include:

[0110] Acquire multiple groups of training samples, wherein each group of training samples includes a sample image, sample image description information of the sample image, and a target part area image in the sample image;

[0111] For each set of training samples, perform the following steps:

[0112] Inputting the sample image description information into a first to-be-trained text encoder, extracting text features in the sample image description information based on the first to-be-trained text encoder, and obtaining a first sample text feature vector corresponding to the sample image description information;

[0113] Inputting the sample image into a first image encoder to be trained, encoding the sample image based on the first image encoder to be trained, and generating a first sample image feature matrix corresponding to the sample image;

[0114] Adding random noise to the first sample image feature matrix to obtain the first sample image feature matrix after adding noise;

[0115] Inputting the first sample text feature vector and the first sample image feature matrix after adding noise into the diffusion model to be trained, and performing denoising processing on the first sample image feature matrix after adding noise by using the first sample text feature vector based on the diffusion model to be trained to obtain the first sample image feature matrix after removing noise;

[0116] Inputting the first sample image feature matrix after noise removal into the first image decoder to be trained, and decoding the first sample image feature matrix after noise removal based on the first image decoder to obtain a predicted sample image;

[0117] Inputting the target part region image into a second image encoder to be trained, encoding the target part region image based on the second image encoder to be trained, and generating a second sample image feature matrix corresponding to the target part region image;

[0118] Adding random noise to the second sample image feature matrix to obtain the second sample image feature matrix after adding noise;

[0119] The second sample text feature vector and the second sample image feature matrix after adding noise are input into the target part reconstruction model to be trained, and the second sample image feature matrix after adding noise is denoised by using the second sample text feature vector based on the target part reconstruction model to be trained, so as to obtain the second sample image feature matrix after removing noise;

[0120] Inputting the feature matrix of the second sample image after the noise is removed into the second image decoder to be trained, decoding the feature matrix of the second sample image after the noise is removed based on the second image decoder to obtain a predicted target part image;

[0121] Based on the predicted sample image, the predicted target part image, the sample image and the target part area image, a target loss function is obtained;

[0122] Based on the target loss function, the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder to be trained, the second text encoder to be trained, the second image encoder to be trained, the target part reconstruction model to be trained, and the second image decoder to be trained are trained to obtain a text encoder, image encoder, diffusion model and image decoder for generating a first image, as well as a text encoder, image encoder, target part reconstruction model and image decoder for generating a second image.

[0123] Among them, the training samples can be used to train the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder to be trained, the second text encoder to be trained, the second image encoder to be trained, the target part reconstruction model to be trained, and the second image decoder to be trained, so as to obtain samples of the text encoder, image encoder, diffusion model and image decoder for generating the first image, and the text encoder, image encoder, target part reconstruction model and image decoder for generating the second image.

[0124] The sample image may be a complete person image including the target part.

[0125] The sample image description information may be description information used to describe the sample image.

[0126] The target part region image may be an image containing only the target part extracted from the sample image. Specifically, the target part in the sample image may be detected by an image detection algorithm to obtain a detection frame of the target part in the sample image, and then the target part in the detection frame may be cropped out to obtain the target part region image in the sample image. In this way, there is no need to obtain the target part region image separately from other platforms or databases, thereby improving the efficiency of obtaining training samples.

[0127] The first text encoder to be trained may be a text encoder to be trained for generating the first image, and after the training of the first text encoder to be trained is completed, the text encoder for generating the first image may be obtained. Correspondingly, the first image encoder to be trained, the diffusion model to be trained, and the first image decoder to be trained may be an image encoder, a diffusion model, and an image decoder to be trained for generating the first image, respectively, and after the training of the first image encoder to be trained, the diffusion model to be trained, and the first image decoder to be trained are completed, the image encoder, the diffusion model, and the image decoder for generating the first image may be obtained, respectively.

[0128] Correspondingly, the second text encoder to be trained can be a text encoder to be trained for generating the second image, and after the training of the second text encoder to be trained is completed, the above-mentioned text encoder for generating the second image can be obtained. Correspondingly, the second image encoder to be trained, the target part reconstruction model to be trained, and the second image decoder to be trained can be an image encoder, a target part reconstruction model, and an image decoder to be trained for generating the second image, respectively. After the training of the second image encoder to be trained, the target part reconstruction model to be trained, and the second image decoder to be trained is completed, the above-mentioned image encoder, target part reconstruction model, and image decoder for generating the second image can be obtained, respectively.

[0129] The first sample text feature vector may be a feature vector corresponding to the sample image description information obtained after the first to-be-trained text encoder extracts text features in the sample image description information.

[0130] The first sample image feature matrix may be a feature matrix obtained by extracting features of the sample image using the first image encoder to be trained.

[0131] The predicted sample image may be a predicted sample image.

[0132] Correspondingly, the second sample text feature vector may be a feature vector corresponding to the sample target part description information obtained after the second to-be-trained text encoder extracts text features in the sample target part description information.

[0133] The sample target part description information mentioned above may be description information containing only the target part extracted from the sample image description information.

[0134] The second sample image feature matrix may be a feature matrix obtained by extracting features of the sample image using the second image encoder to be trained.

[0135] The predicted target part image may be an image of the predicted target part.

[0136] The target loss function can be the loss function of the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder to be trained, the second text encoder to be trained, the second image encoder to be trained, the target part reconstruction model to be trained, and the second image decoder to be trained.

[0137] In some embodiments of the present application, text features in sample image description information can be extracted based on a first text encoder to be trained to obtain a first sample text feature vector corresponding to the sample image description information; features of the sample image can be extracted based on the first image encoder to be trained to generate a first sample image feature matrix corresponding to the sample image; random noise is added to the first sample image feature matrix to obtain a first sample image feature matrix after noise is added; the first sample image feature matrix after noise is added is denoised using the first sample text feature vector based on a diffusion model to be trained to obtain a first sample image feature matrix after noise is removed; the first sample image feature matrix after noise is removed is decoded based on the first image decoder to be trained to obtain a predicted sample image.

[0138] Extract the description information of the target part from the sample image description information to obtain the sample target part description information, and extract the target part from the sample image to obtain the target part area image, and then input the sample target part description information into the second text encoder to be trained, and extract the text features in the sample target part description information based on the second text encoder to be trained to obtain the second sample text feature vector; extract the features of the target part area image based on the second image encoder to be trained to generate a second sample image feature matrix corresponding to the target part area image; add random noise to the second sample image feature matrix to obtain the second sample image feature matrix after adding noise; denoise the second sample image feature matrix after adding noise using the second sample text feature vector based on the target part reconstruction model to obtain the second sample image feature matrix after removing noise; decode the second sample image feature matrix after removing noise based on the second image decoder to obtain the predicted target part image.

[0139] According to the predicted sample image, sample image, predicted target part image, and target part area image, the target loss function of the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder to be trained, the second text encoder to be trained, the second image encoder to be trained, the target part reconstruction model to be trained, and the second image decoder to be trained can be determined. Then, based on the target loss function, the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder to be trained, the second text encoder to be trained, the second image encoder to be trained, the target part reconstruction model to be trained, and the second image decoder to be trained can be trained to obtain a text encoder, an image encoder, a diffusion model, and an image decoder for generating a first image, as well as a text encoder, an image encoder, a target part reconstruction model, and an image decoder for generating a second image.

[0140] In an embodiment of the present application, a first text encoder to be trained, a first image encoder to be trained, a diffusion model to be trained, a first image decoder to be trained, a second text encoder to be trained, a second image encoder to be trained, a target part reconstruction model to be trained, and a second image decoder to be trained are trained using training samples to obtain a text encoder, an image encoder, a diffusion model and an image decoder for generating a first image, as well as a text encoder, an image encoder, a target part reconstruction model and an image decoder for generating a second image, so that the corresponding image can be subsequently obtained based on the text encoder, image encoder, diffusion model and image decoder for generating the first image, and the text encoder, image encoder, target part reconstruction model and image decoder for generating the second image, so as to improve the generation efficiency and accuracy of the image corresponding to the image description information or the image corresponding to the target part description information.

[0141] In some embodiments of the present application, the above-mentioned target loss function may be composed of three parts, that is, the target loss function may include a first loss function, a second loss function and a third loss function, wherein the first loss function may be used to characterize the error between the predicted sample image and the sample image, the second loss function may be used to characterize the error between the predicted target part image and the target part area image, and the third loss function may be used to characterize the error between the sample image and the image obtained after replacing the target part in the predicted sample image with the predicted target part image.

[0142] In order to improve the image quality of the third image, obtaining the target loss function based on the predicted target image, the sample image and the target part area image may specifically include:

[0143] Based on the predicted sample image and the sample image, a first loss function is obtained;

[0144] Based on the predicted target part image and the target part region image, a second loss function is obtained;

[0145] The target part region in the predicted sample image is replaced with the predicted target part image to obtain a predicted image;

[0146] Based on the predicted image and the sample image, a third loss function is obtained;

[0147] The sum of the first loss function, the second loss function and the third loss function is taken as the target loss function.

[0148] The predicted image may be an image obtained by replacing the target part region in the predicted sample image with the predicted target part image.

[0149] In some embodiments of the present application, a first loss function may be obtained based on the predicted sample image and the sample image. Specifically, a mean square error loss function between the predicted sample image and the sample image may be calculated and used as the first loss function.

[0150] Based on the predicted target part image and the target part region image, a second loss function is obtained. Specifically, a mean square error loss function between the predicted target part image and the target part region image is calculated and used as the second loss function.

[0151] Based on the predicted image and the sample image, a third loss function is obtained, specifically, a perceptual loss function between the predicted image and the sample image is calculated and used as the third loss function. The perceptual loss function here can be the direct feature difference between the calculated generated image and the real image, which can better reflect human perception of the image.

[0152] Then the sum of the first loss function, the second loss function and the third loss function is taken as the target loss function.

[0153] That is, the target loss function can be calculated according to the following formula (1):

[0154] Loss = MSE(I,P) + MSE(I hand ,P hand )+MSE(I,P+P hand ) (1)

[0155] Among them, I is the sample image, P is the predicted sample image, I hand is the target area image, P hand To predict the target part image, MSE is the mean square error loss function and PL is the perceptual loss function.

[0156] In an embodiment of the present application, by respectively calculating the error between the predicted sample image and the sample image, the error between the predicted target part image and the target part area image, the error between the image obtained after replacing the target part in the predicted sample image with the predicted target part image and the sample image, the model can not only pay attention to the quality of the reconstruction of the target part area, but also guide the degree of fit between the separately generated target part area and the first image, so as to obtain a third image in which the target part is neither deformed nor inconsistent, thereby improving the image quality of the third image.

[0157] The following introduces the internal processing flow of the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder to be trained, the second text encoder to be trained, the second image encoder to be trained, the target part reconstruction model to be trained, and the second image decoder to be trained. Since the internal training processes of the first text encoder to be trained and the second text encoder to be trained are consistent, the internal training processes of the first image encoder to be trained and the second image encoder to be trained are consistent, the internal training processes of the diffusion model to be trained and the target part reconstruction model to be trained are consistent, and the internal training processes of the first image decoder to be trained and the second image decoder to be trained are consistent, the internal processing flow of the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, and the first image decoder to be trained are taken as an example to illustrate the internal training process of the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder, the second text encoder to be trained, the second image encoder, the target part reconstruction model to be trained, and the second image decoder to be trained.

[0158] 1. The first text encoder to be trained

[0159] In some embodiments of the present application, the first text encoder to be trained may include an encoding layer and a transformation layer; and the corresponding second text editor to be trained also includes an encoding layer and a transformation layer.

[0160] The extracting text features in the sample image description information based on the first to-be-trained text encoder to obtain a first sample text feature vector corresponding to the sample image description information may specifically include:

[0161] Performing word segmentation processing on the sample image description information based on the coding layer to obtain at least one sample target word segmentation;

[0162] Determine, based on the encoding layer, a sample target word segmentation vector corresponding to at least one sample target word segmentation according to at least one sample target word segmentation and a pre-set correspondence relationship between each word segmentation and the word segmentation vector corresponding to each word segmentation;

[0163] Determine a first sample text representation vector corresponding to the sample image description information based on the sample target word segmentation vectors respectively corresponding to at least one sample target word segmentation;

[0164] Based on the transformation layer, multi-layer self-attention calculation is performed on the first sample text representation vector to obtain the first sample text feature vector corresponding to the sample image description information.

[0165] The sample target word segmentation may be the word segmentation obtained after word segmentation processing is performed on the sample image description information.

[0166] The sample target word segmentation vector may be a word segmentation vector corresponding to the sample target word segmentation.

[0167] The first sample text representation vector may be a vector corresponding to the sample image description information obtained based on the sample target word segmentation vectors corresponding to at least one sample target word segmentation. Specifically, it may be a vector obtained by sorting the sample target word segmentation vectors corresponding to at least one sample target word segmentation in a certain order.

[0168] In some embodiments of the present application, the sample image description information can be segmented based on the encoding layer to obtain at least one sample target segmentation, and then based on the encoding layer, according to the at least one sample target segmentation and the correspondence between each pre-set segmentation and the segmentation vector corresponding to each segmentation, a sample target segmentation vector corresponding to at least one sample target segmentation is determined; based on the sample target segmentation vector corresponding to at least one sample target segmentation, a first sample text representation vector corresponding to the sample image description information is determined; based on the transformation layer, multi-layer self-attention calculation is performed on the first sample text representation vector to obtain a first sample text feature vector corresponding to the sample image description information.

[0169] Continuing with the above example, refer to Figure 6, Figure 6 : is a flow chart of the internal processing process of the first text encoder to be trained. The sample image description information "a girl wearing a skirt with her right hand on her left shoulder" is input into the encoding layer. The encoding layer uses Tokenizer encoding to perform word segmentation processing on the sample image description information "a girl wearing a skirt with her right hand on her left shoulder" to obtain at least one sample target word segmentation. Then, the encoding layer determines the sample target word segmentation vector corresponding to at least one sample target word segmentation according to the at least one sample target word segmentation and the correspondence between each pre-set word segmentation and the word segmentation vector corresponding to each word segmentation. Then, based on the sample target word segmentation vector corresponding to the at least one sample target word segmentation, a first sample text representation vector corresponding to the sample image description information is obtained. The dimension of the first sample text representation vector is [128,768], where 128 is the maximum number of dimensions supported by the first text encoder to be trained, and 768 is the vector length corresponding to each dimension.

[0170] The first sample text representation vector is input into the transformation layer, where the transformation layer can be a Transformer module. Multi-layer self-attention calculation is performed on the first sample text representation vector based on the transformation layer to obtain the first sample text feature vector TextEmbedding corresponding to the sample image description information. The dimension of the first sample text feature vector TextEmbedding is [128,768]. The first sample text feature vector captures the semantic information of the sample image description information and can be used as a vector expression of the sample image description information.

[0171] It should be noted that in Figure 6 In the transformation layer, the self-attention operator SelfAttention, the addition operator Add, the normalization operator Norm, and the forward feedback operator Feed Forward can be included in the transformation layer. In addition, Figure 6 The transformation layers in are stacked up to 36 layers.

[0172] In the second text encoder to be trained, the transformation layer is stacked into 12 layers, such as Figure 7 As shown in Figure 1, this is because the sample target part description information is shorter than the sample image description information, and compared to a large text editor, Figure 7 The second text editor to be trained shown will not only not affect the effect, but also greatly reduce the computing power consumption. Figure 6 The first text editor to be trained in the training stacks 36 layers of transformation layers, which has more parameters, stronger modeling energy and expression ability, and can handle more complex language descriptions. Figure 6The first text editor to be trained can achieve better image generation results.

[0173] 2. The first image encoder to be trained

[0174] In some embodiments of the present application, the first image encoder to be trained may include a convolution module and a transformation module; the processing of the sample image based on the first image encoder to be trained to generate a first sample image feature matrix corresponding to the sample image may specifically include:

[0175] Extracting image features from the sample image based on the convolution module to obtain a first sample feature matrix corresponding to the sample image;

[0176] Obtaining the dependency relationship between the elements in the first sample feature matrix based on the transformation module to obtain a second sample feature matrix;

[0177] Performing feature extraction on the second sample feature matrix based on the convolution module to obtain a third sample feature matrix;

[0178] Obtaining the dependency relationship between the elements in the third sample feature matrix based on the transformation module to obtain a fourth sample feature matrix;

[0179] Based on the convolution module, feature extraction is performed on the fourth sample feature matrix to generate a first sample image feature matrix corresponding to the sample image.

[0180] The convolution module may be a module for extracting image features, and the convolution module may have a convolution operator, such as Conv.

[0181] The transformation module may be a module for capturing dependencies between data in a feature matrix, for example, the transformation module may be a Transformer.

[0182] The first sample feature matrix may be obtained by performing image feature extraction on the sample image based on a convolution module to obtain a feature matrix corresponding to the sample image.

[0183] The second sample feature matrix can be a feature matrix obtained after the transformation module obtains the dependency relationship between the elements in the first sample feature matrix.

[0184] The third sample feature matrix may be a feature matrix obtained after the convolution module further extracts features from the second sample feature matrix.

[0185] The fourth sample feature matrix may be a feature matrix obtained after the transformation module obtains the dependency relationship between the elements in the third sample feature matrix.

[0186] In some embodiments of the present application, image features can be extracted from a sample image based on a convolution module to obtain a first sample feature matrix corresponding to the sample image; the dependency relationship between the elements in the first sample feature matrix can be obtained based on a transformation module to obtain a second sample feature matrix; features are extracted from the second sample feature matrix based on the convolution module to obtain a third sample feature matrix; the dependency relationship between the elements in the third sample feature matrix can be obtained based on the transformation module to obtain a fourth sample feature matrix; features are extracted from the fourth sample feature matrix based on the convolution module to generate a first sample image feature matrix corresponding to the sample image.

[0187] Continuing with the above example, refer to Figure 8 The resolution of the sample image is 512*512*3. It is input into the convolution module 1 for calculation and preliminary feature compression to obtain a feature matrix with a dimension of [4, 256, 256], which is the first sample feature matrix.

[0188] Then, the first sample feature matrix with a dimension of [4, 256, 256] is input into the transformation module 1. Based on the dependence between the elements in the first sample feature matrix captured by the transformation module 1, the feature matrix with a dimension of [4, 256, 256] is output, i.e., the second sample feature matrix. The transformation module 1 here can be a Transformer module, which can be stacked with 4 layers to increase the fitting energy of the first image encoder to be trained. Its structure is the same as the Transformer structure in the first text encoder to be trained.

[0189] Then the second sample feature matrix with a dimension of [4, 256, 256] is further passed through the convolution module 2 to output the third sample feature matrix of [8, 128, 128], the third sample feature matrix of [8, 128, 128] is passed through the transformation module 2 to output the fourth sample feature matrix of [8, 128, 128], and the fourth sample feature matrix of [8, 128, 128] is passed through the convolution module 3 to finally obtain the first sample image feature matrix with a dimension of [16, 64, 64].

[0190] It should be noted that the function of convolution module 1, convolution module 2 and convolution module 3 is to reduce the feature resolution. The only difference among them is the number of convolution kernels. Specifically, the number of channels of convolution module 1 is 4, the number of channels of convolution module 2 is 8, and the number of channels of convolution module 3 is 16.

[0191] In the embodiment of the present application, the sample image can be processed by the convolution module and the transformation module in the first image encoder to be trained to accurately generate a first sample image feature matrix corresponding to the sample image.

[0192] In some embodiments of the present application, the convolution module in the first image encoder to be trained may include a first convolution layer, a first non-linear function activation layer and a second convolution layer, that is, the convolution module 1, the convolution module 2 and the convolution module 3 in the first image encoder to be trained may all include a first convolution layer, a first non-linear function activation layer and a second convolution layer.

[0193] The step of extracting image features from the sample image based on the convolution module to obtain a first sample feature matrix corresponding to the sample image may specifically include:

[0194] Performing image feature extraction on the sample image based on the first convolutional layer to obtain a fifth sample feature matrix;

[0195] Performing nonlinear function activation on the fifth sample feature matrix based on the first nonlinear function activation layer to obtain a sixth sample feature matrix;

[0196] Performing feature extraction on the sixth sample feature matrix based on the second convolutional layer to obtain a seventh sample feature matrix;

[0197] Performing nonlinear function activation on the seventh sample feature matrix based on the first nonlinear function activation layer to obtain an eighth sample feature matrix;

[0198] Performing feature extraction on the eighth sample feature matrix based on the second convolutional layer to obtain a ninth sample feature matrix;

[0199] Based on the first nonlinear function activation layer, the ninth sample feature matrix is ​​activated by a nonlinear function to obtain a first sample feature matrix corresponding to the sample target image.

[0200] The fifth sample feature matrix may be a feature matrix obtained after performing image feature extraction on the sample image based on the first convolutional layer.

[0201] The sixth sample feature matrix may be a feature matrix obtained by performing nonlinear function activation on the fifth sample feature matrix based on the first nonlinear function activation layer.

[0202] The seventh sample feature matrix may be a feature matrix obtained after extracting features from the sixth sample feature matrix based on the second convolutional layer.

[0203] The eighth sample feature matrix may be a feature matrix obtained by performing nonlinear function activation on the seventh sample feature matrix based on the first nonlinear function activation layer;

[0204] The ninth sample feature matrix may be a feature matrix obtained after extracting features from the eighth sample feature matrix based on the second convolutional layer.

[0205] In some embodiments of the present application, image features can be extracted from the sample image based on the first convolution layer to obtain a fifth sample feature matrix; nonlinear function activation can be performed on the fifth sample feature matrix based on the first nonlinear function activation layer to obtain a sixth sample feature matrix; feature extraction can be performed on the sixth sample feature matrix based on the second convolution layer to obtain a seventh sample feature matrix; nonlinear function activation can be performed on the seventh sample feature matrix based on the first nonlinear function activation layer to obtain an eighth sample feature matrix; feature extraction can be performed on the eighth sample feature matrix based on the second convolution layer to obtain a ninth sample feature matrix; nonlinear function activation can be performed on the ninth sample feature matrix based on the first nonlinear function activation layer to obtain a first sample feature matrix corresponding to the sample image.

[0206] Continue to refer Figure 8 , the sample image with a resolution of 512*512*3 is input into the convolution layer 1 in the convolution module 1 to calculate the fifth sample feature matrix of [4,256,256], the fifth sample feature matrix of [4,256,256] is input into the nonlinear function activation layer 1 for nonlinear function activation, and the sixth sample feature matrix with a dimension of [4,256,256] is output, and the sixth sample feature matrix with a dimension of [4,256,256] is further input into the convolution layer 2 in the convolution module 1 to calculate the seventh feature matrix of [4,256,256], and the seventh sample feature matrix of [4,256,256] is input into the nonlinear function activation layer 1 for nonlinear function activation, and the eighth sample feature matrix with a dimension of [4,256,256] is output. The eighth sample feature matrix with a dimension of [4, 256, 256] is further input into the convolution layer 3 in the convolution module 1 to obtain the ninth sample feature matrix of [4, 256, 256], and the ninth sample feature matrix of [4, 256, 256] is input into the nonlinear function activation layer 1 for nonlinear function activation, and the first sample feature matrix with a dimension of [4, 256, 256] is output.

[0207] It should be noted that the convolution kernel size of convolution layer 1 is 3*3, the step size is 2, the number of convolution kernels is 4, and the step sizes of convolution layer 2 and convolution layer 3 are both 1 to ensure that the dimension of the output feature matrix is ​​still [4, 256, 256].

[0208] In an embodiment of the present application, image features of a sample image are extracted through the first convolution layer, the first nonlinear function activation layer, and the second convolution layer in the convolution module, so that a first sample feature matrix corresponding to the sample image can be accurately obtained.

[0209] 3. The first image decoder to be trained

[0210] In some embodiments of the present application, the first image decoder to be trained may include a deconvolution module and an inverse transformation module; the decoding of the first sample image feature matrix after noise removal based on the first image decoder to obtain a predicted sample image may include:

[0211] Performing feature decoding on the first sample image feature matrix after noise removal based on the deconvolution module to obtain a fourth sample feature matrix;

[0212] Obtaining the dependency relationship between the elements in the fourth sample feature matrix based on the inverse transformation module to obtain the third sample feature matrix;

[0213] Performing feature extraction on the third sample feature matrix based on the deconvolution module to obtain a second sample feature matrix;

[0214] Obtain the dependency relationship between the elements in the second sample feature matrix based on the inverse transformation module to obtain the first sample feature matrix;

[0215] Based on the deconvolution module, feature extraction is performed on the first sample feature matrix to generate a predicted sample image.

[0216] The deconvolution module may be a module for decoding the features of the image after convolution, and the deconvolution module may have a deconvolution operator, such as Deconv.

[0217] The inverse transform module may be a module for decoding a matrix transformed by the transform module.

[0218] In some embodiments of the present application, after the sample image is convolved and transformed using the first image to be trained encoder as mentioned above, the feature matrix of the first sample image after noise removal can be feature decoded based on the deconvolution module in the first image to be trained decoder to obtain a fourth sample feature matrix, the dependency relationship between the elements in the fourth sample feature matrix is ​​obtained based on the inverse transformation module to obtain a third sample feature matrix, the feature extraction of the third sample feature matrix is ​​performed based on the deconvolution module to obtain a second sample feature matrix, the dependency relationship between the elements in the second sample feature matrix is ​​obtained based on the inverse transformation module to obtain a first sample feature matrix, the feature extraction of the first sample feature matrix is ​​performed based on the deconvolution module to generate a predicted sample image.

[0219] Continuing with the above example, refer to Fig. 9, the first sample image feature matrix with a dimension of [16, 64, 64] is preliminarily decoded by the deconvolution module 1 to obtain a fourth sample feature matrix with a dimension of [8, 128, 128], and then the fourth sample feature matrix with a dimension of [8, 128, 128] is passed through the inverse transformation module 1 to obtain the dependency between the elements in the fourth sample feature matrix, and the third sample feature matrix of [8, 128, 128] is output, and the third sample feature matrix of [8, 128, 128] is passed through the deconvolution module 2 to obtain a second sample feature matrix of [4, 256, 256], and the second sample feature matrix of [4, 256, 256] is passed through the inverse transformation module 1 to obtain a first sample feature matrix of [4, 256, 256], and the first sample feature matrix of [4, 256, 256] is passed through the deconvolution module 3 to obtain a predicted sample image of [3, 512, 512].

[0220] It should be noted that the function of deconvolution module 1, deconvolution module 2 and deconvolution module 3 is to increase feature resolution. The only difference among the three is the number of channels of the convolution kernel. Specifically, the number of channels of deconvolution module 1 is 16, the number of channels of deconvolution module 2 is 8, and the number of channels of deconvolution module 3 is 4.

[0221] In the embodiment of the present application, the first sample image feature matrix after noise removal is decoded by the deconvolution module and the inverse transformation module in the first image decoder to be trained, so as to accurately obtain the predicted sample image.

[0222] In some embodiments of the present application, the deconvolution module in the above-mentioned first image decoder to be trained may include a first deconvolution layer, a second non-linear function activation layer and a second deconvolution layer, that is, the above-mentioned deconvolution module 1, deconvolution module 2 and deconvolution module 3 may all include a first deconvolution layer, a second non-linear function activation layer and a second deconvolution layer.

[0223] The step of performing feature decoding on the first sample image feature matrix after noise removal based on the deconvolution module to obtain a fourth sample feature matrix may include:

[0224] Performing feature extraction on the first sample image feature matrix after noise removal based on the first deconvolution layer to obtain a tenth sample feature matrix;

[0225] Based on the second nonlinear function activation layer, a nonlinear function is performed on the tenth sample feature matrix to obtain an eleventh sample feature matrix;

[0226] Performing feature extraction on the eleventh sample feature matrix based on the second deconvolution layer to obtain a twelfth sample feature matrix;

[0227] Performing nonlinear function activation on the twelfth sample feature matrix based on the second nonlinear function activation layer to obtain a thirteenth sample feature matrix;

[0228] Performing feature extraction on the thirteenth sample feature matrix based on the second deconvolution layer to obtain a fourteenth sample feature matrix;

[0229] Based on the second nonlinear function activation layer, the fourteenth sample feature matrix is ​​activated by a nonlinear function to obtain a fourth sample feature matrix.

[0230] The tenth sample feature matrix may be a feature matrix obtained by extracting features from the first sample image feature matrix after noise removal based on the first deconvolution layer.

[0231] The eleventh sample feature matrix may be a feature matrix obtained by performing nonlinear function activation on the tenth sample feature matrix based on the second nonlinear function activation layer.

[0232] The twelfth sample feature matrix may be a feature matrix obtained after the second deconvolution layer performs feature extraction on the eleventh sample feature matrix.

[0233] The thirteenth sample feature matrix may be a feature matrix obtained by performing nonlinear function activation on the twelfth sample feature matrix based on the second nonlinear function activation layer.

[0234] The fourteenth sample feature matrix may be a feature matrix obtained after extracting features from the thirteenth sample feature matrix based on the second deconvolution layer.

[0235] In some embodiments of the present application, feature extraction can be performed on the first sample image feature matrix after noise removal based on the first deconvolution layer to obtain a tenth sample feature matrix, and nonlinear function activation can be performed on the tenth sample feature matrix based on the second nonlinear function activation layer to obtain an eleventh sample feature matrix; feature extraction can be performed on the eleventh sample feature matrix based on the second deconvolution layer to obtain a twelfth sample feature matrix; nonlinear function activation can be performed on the twelfth sample feature matrix based on the second nonlinear function activation layer to obtain a thirteenth sample feature matrix; feature extraction can be performed on the thirteenth sample feature matrix based on the second deconvolution layer to obtain a fourteenth sample feature matrix; nonlinear function activation can be performed on the fourteenth sample feature matrix based on the second nonlinear function activation layer to obtain a fourth sample feature matrix.

[0236] Continue to refer Fig. 9, the first sample image feature matrix with a dimension of [16, 64, 64] is initially decoded by the deconvolution layer 1 in the deconvolution module 1 to increase the feature resolution, and the tenth sample feature matrix with a dimension of [8, 128, 128] is obtained. The tenth sample feature matrix with a dimension of [8, 128, 128] is activated by the nonlinear function activation layer 1 to obtain the eleventh sample feature matrix with a dimension of [8, 128, 128]. The eleventh sample feature matrix with a dimension of [8, 128, 128] is extracted by the deconvolution layer 2 in the deconvolution module 1 to obtain the twelfth sample feature matrix with a dimension of [8, 128, 128]. The twelfth sample feature matrix with a dimension of [8, 128, 128] is activated by a nonlinear function through a nonlinear function activation layer 1 to obtain a thirteenth sample feature matrix with a dimension of [8, 128, 128]. The thirteenth sample feature matrix with a dimension of [8, 128, 128] is extracted by a deconvolution layer 3 in a deconvolution module 1 to obtain a fourteenth sample feature matrix with a dimension of [8, 128, 128]. The fourteenth sample feature matrix with a dimension of [8, 128, 128] is activated by a nonlinear function through a nonlinear function activation layer 1 to obtain a fourth sample feature matrix with a dimension of [8, 128, 128].

[0237] It should be noted that the deconvolution kernel size of deconvolution layer 1 is 3*3, the stride is 2, the number of output channels is 8, and the stride of deconvolution layer 2 and deconvolution layer 3 is 1 to keep the feature dimension unchanged [8, 128, 128].

[0238] In an embodiment of the present application, the first deconvolution layer, the second nonlinear function activation layer and the second deconvolution layer in the deconvolution module are used to perform feature decoding on the first sample image feature matrix after noise removal, so as to accurately obtain the fourth sample feature matrix.

[0239] 4. Diffusion Model to be Trained

[0240] In some embodiments of the present application, the diffusion model to be trained may include an input processing module, a noise prediction module and a noise removal module. The corresponding target part reconstruction model to be trained also includes an input processing module, a noise prediction module and a noise removal module.

[0241] The above-mentioned input processing module may be a module for processing the input of the diffusion model to be trained. The noise prediction module may be a module for predicting the noise distribution in the feature matrix of the first sample image after adding noise, and the specific noise prediction module may be DiT. The noise removal module may be a module for removing the noise in the feature matrix of the first sample image after adding noise.

[0242] The method of performing denoising processing on the first sample image feature matrix after adding noise by using the first sample text feature vector based on the diffusion model to be trained to obtain the first sample image feature matrix after removing noise may include:

[0243] Preprocessing the first sample text feature vector and the first sample image feature matrix after adding noise based on the input processing module to obtain a first sample attention embedding feature vector;

[0244] Predicting the noise distribution in the first sample attention embedding feature vector based on the noise prediction module;

[0245] The noise distributed in the first sample image feature matrix after adding noise is removed based on the noise removal module to obtain the first sample image feature matrix after removing noise.

[0246] Among them, the first sample attention embedding feature vector can be a feature vector obtained after preprocessing the first sample text feature vector and the first sample image feature matrix after adding noise based on the input processing module.

[0247] In some embodiments of the present application, the first sample text feature vector and the first sample image feature matrix after adding noise can be preprocessed based on the input processing module to obtain the first sample attention embedding feature vector, and then the noise distribution in the first sample attention embedding feature vector is predicted based on the noise prediction module. Finally, the noise distributed in the first sample image feature matrix after adding noise is removed based on the noise removal module to obtain the first sample image feature matrix after removing the noise. The specific processes of the input processing module, the noise prediction module and the noise removal module will be introduced in detail in the subsequent embodiments.

[0248] In an embodiment of the present application, the first sample text feature vector and the first sample image feature matrix after adding noise are preprocessed by an input processing module to obtain a first sample attention embedding feature vector, and then the noise distribution in the first sample attention embedding feature vector is predicted based on a noise prediction module. Finally, the noise distributed in the first sample image feature matrix after adding noise is removed based on a noise removal module, so that the first sample image feature matrix after removing noise can be accurately obtained.

[0249] In some embodiments of the present application, the above-mentioned input processing module may include: a linear layer, a blocking layer and a splicing layer; the linear layer here may be a layer for performing feature mapping on feature vectors, and the linear layer may include a linear transformation operator Linear for mapping the data space.

[0250] The input processing module preprocesses the first sample text feature vector and the first sample image feature matrix after adding noise to obtain the first sample attention embedding feature vector, including:

[0251] Perform feature mapping on the first sample text feature vector based on a linear layer to obtain a first sample mapping feature vector;

[0252] Based on the block layer, a block operation is performed on the feature matrix of the first sample image after adding noise to obtain at least one feature block;

[0253] Perform feature mapping on the feature vectors corresponding to at least one feature block based on the linear layer to obtain at least one second sample mapping feature vector;

[0254] The first mapping feature vector and at least one second mapping feature vector are concatenated based on the concatenation layer to obtain a first sample attention embedding feature vector.

[0255] The first sample mapping feature vector may be a feature vector obtained after feature mapping is performed on the first sample text feature vector based on a linear layer.

[0256] The second sample mapping feature vector may be a feature vector obtained by performing feature mapping on feature vectors corresponding to at least one feature block based on a linear layer.

[0257] Continuing with the above example, refer to Fig.10 and Fig.11 , Fig.10 It is a schematic diagram of the internal processing flow of the diffusion model to be trained. Fig.11 Schematic diagram of the content processing flow for reconstructing the model of the target part to be trained. Fig.10 In the method, a linear layer is used to perform feature mapping on the feature vector of the first sample text to obtain a first sample mapping feature vector, and then a blocking layer is used to perform a blocking operation on the feature matrix of the first sample image after adding noise to obtain at least one feature block, and then a linear layer is used to perform feature mapping on the feature vector corresponding to the at least one feature block to obtain at least one second sample mapping feature vector, and then the first sample mapping feature vector and the at least one second sample mapping feature vector are spliced ​​based on the splicing layer to obtain a first sample attention embedding feature vector.

[0258] exist Fig.11In the method, the second sample text feature vector is feature mapped using a linear layer to obtain a third mapping feature vector, and then the second sample image feature matrix after adding noise is partitioned using a blocking layer to obtain at least one feature block, and then the feature vectors corresponding to the at least one feature block are feature mapped using a linear layer to obtain at least one fourth mapping feature vector, and then the third sample mapping feature vector and at least one fourth sample mapping feature vector are spliced ​​based on the splicing layer to obtain the second sample attention embedding feature vector.

[0259] The third sample mapping feature vector mentioned above may be a feature vector obtained after feature mapping the second sample text feature vector based on a linear layer.

[0260] The fourth sample mapping feature vector may be a feature vector obtained by performing feature mapping on feature vectors corresponding to at least one feature block based on a linear layer.

[0261] The second sample attention embedding feature vector may be a feature vector obtained by preprocessing the second sample text feature vector and the second sample image feature matrix after adding noise based on the input processing module in the target part reconstruction model to be trained.

[0262] In an embodiment of the present application, the first sample text feature vector and the first sample image feature matrix after adding noise can be preprocessed through the linear layer, blocking layer, encoding layer and splicing layer in the input processing module to accurately obtain the first sample attention embedding feature vector.

[0263] In some embodiments of the present application, the input processing module may further include an encoding layer. Before the first sample mapping feature vector and at least one second sample mapping feature vector are concatenated based on the concatenation layer to obtain the first sample attention embedding feature vector, the method may further include:

[0264] Performing position encoding on at least one second mapping feature vector based on the encoding layer to obtain at least one second sample mapping feature vector after position encoding;

[0265] The step of concatenating the first sample mapping feature vector and at least one second sample mapping feature vector based on the concatenation layer to obtain the first sample attention embedding feature vector may include:

[0266] Based on the concatenation layer, the first sample mapping feature vector and at least one second sample mapping feature vector after position encoding are concatenated to obtain a first sample attention embedding feature vector.

[0267] In some embodiments of the present application, at least one second sample mapping feature vector can be position-encoded based on the encoding layer to obtain at least one second sample mapping feature vector after position encoding, and then the first sample mapping feature vector and the at least one second sample mapping feature vector after position encoding are spliced ​​based on the splicing layer to obtain the first sample attention embedding feature vector. The role of the position encoding here is to mark the position order of each feature block.

[0268] In the target part reconstruction model to be trained, rotation position encoding can be performed on at least one fourth sample mapping feature vector based on the encoding layer in the target part reconstruction model to be trained, so as to obtain at least one fourth sample mapping feature vector after rotation position encoding;

[0269] Based on the concatenation layer, the third sample mapping feature vector and at least one fourth sample mapping feature vector after rotation position encoding are concatenated to obtain a second sample attention embedding feature vector.

[0270] In the reconstruction model of the target part to be trained, compared with the position encoding in the above-mentioned diffusion model to be trained, the rotational position encoding can improve the performance of the reconstruction model of the target part to be trained and reduce information loss, and is more advantageous in processing delicate work such as target part images.

[0271] In an embodiment of the present application, at least one second sample mapping feature vector may be encoded based on the encoding layer, thereby improving the training flexibility of the diffusion model to be trained.

[0272] In some embodiments of the present application, the noise prediction module in the diffusion model to be trained may also include: a feature scaling layer, a self-attention operation layer, and an opinion feedback layer. The feature scaling layer here may be a layer for scaling feature vectors. The self-attention operation layer may have a multi-head attention calculation (Multi Head SelfAttention, MHSA) operator, and the opinion feedback layer may have a forward feedback operator Feed Forward.

[0273] Predicting the noise distribution in the first sample attention embedding feature vector based on the noise prediction module may specifically include:

[0274] Based on the feature scaling layer, scaling the first sample attention embedding feature vector using a first scaling factor and a second scaling factor in the first noise feature vector to obtain a first sample scaled feature vector;

[0275] Perform a self-attention operation on the first sample scaled feature vector based on the self-attention operation layer to obtain a first sample self-attention feature vector;

[0276] Based on the feature scaling layer, the first sample self-attention feature vector is scaled using a third scaling factor in the first noise feature vector to obtain a second sample scaled feature vector;

[0277] Based on the feature scaling layer, scaling the second sample scaling feature vector using the first scaling factor and the second scaling factor to obtain a third sample scaling feature vector;

[0278] Performing nonlinear calculation on the third sample scaled feature vector based on the opinion feedback layer to obtain a fourth sample scaled feature vector;

[0279] Based on the feature scaling layer, scaling the fourth sample scaling feature vector using a third scaling factor to obtain a fifth sample scaling feature vector;

[0280] Based on the feature scaling layer, residual calculation is performed on the fifth sample scaled feature vector and the second sample scaled feature vector to obtain the noise distribution in the first sample attention embedding feature vector.

[0281] Among them, the first noise feature vector can be a pre-set constant vector used to characterize the noise level in the first sample image feature matrix after adding noise. The first noise feature vector, like the above-mentioned first sample text feature vector and the first sample image feature matrix after adding noise, is the input of the diffusion model to be trained.

[0282] The first scaling factor, the second scaling factor, and the third scaling factor are all scaling parameters in the first noise feature vector.

[0283] The first sample scaled feature vector may be a feature vector obtained by scaling the first sample attention embedding feature vector based on the feature scaling layer using the first scaling factor and the second scaling factor in the first noise feature vector.

[0284] The first sample self-attention feature vector may be a feature vector obtained by performing a self-attention operation on the first sample scaled feature vector based on the self-attention operation layer.

[0285] The second sample scaled feature vector may be a feature vector obtained by scaling the first sample self-attention feature vector based on the feature scaling layer using the third scaling factor in the first noise feature vector.

[0286] The third sample scaled feature vector may be a feature vector obtained by scaling the second sample scaled feature vector using the first scaling factor and the second scaling factor based on the feature scaling layer.

[0287] The fourth sample scaled feature vector is a feature vector obtained by performing nonlinear calculation on the third sample scaled feature vector based on the opinion feedback layer.

[0288] The fifth sample scaled feature vector is a feature vector obtained by scaling the fourth sample scaled feature vector by the feature scaling layer using the third scaling factor.

[0289] Continue to refer Fig.10 , the first scaling factor α learned from the first noise feature vector can be used based on the feature scaling layer x and a second scaling factor β x The first sample attention embedding feature vector is scaled according to the following formula (2) to obtain the first sample scaled feature vector.

[0290] Attention Embedding α x +β x (2)

[0291] Among them, AttentionEmbedding is the first sample attention embedding feature vector.

[0292] Then, based on the self-attention operation layer, the first sample scaled feature vector is self-attention operated to obtain the first sample self-attention feature vector, and then based on the feature scaling layer, the third scaling factor γ in the first noise feature vector is used x The first sample self-attention feature vector is scaled to obtain the second sample scaled feature vector, and the first scaling factor α is used based on the feature scaling layer. x and a second scaling factor β x The second sample scaling feature vector is scaled to obtain a third sample scaling feature vector. The third sample scaling feature vector is nonlinearly calculated based on the opinion feedback layer to obtain a fourth sample scaling feature vector. The third scaling factor γ is used based on the feature scaling layer to obtain a fourth sample scaling feature vector. x The fourth sample scaled feature vector is scaled to obtain a fifth sample scaled feature vector, and the fifth sample scaled feature vector and the second sample scaled feature vector are residually calculated based on the feature scaling layer to obtain the noise distribution in the first sample attention embedding feature vector.

[0293] It should be noted that Fig.10 and Fig.11 The noise prediction modules in are stacked with 16 layers.

[0294] In an embodiment of the present application, the noise distribution in the first sample attention embedding feature vector can be accurately predicted through the feature scaling layer, the self-attention operation layer and the opinion feedback layer in the noise prediction module.

[0295] In some embodiments of the present application, reference Fig.11In the target part reconstruction model to be trained, the noise prediction module may also include: a feature scaling layer, a self-attention operation layer, a linear layer, two normalization layers and an activation layer; the noise distribution in the second sample attention embedding feature vector is predicted based on the noise prediction module, which may include:

[0296] Scaling the second sample attention embedding feature vector using the first scaling factor and the second scaling factor in the second noise feature vector based on the feature scaling layer to obtain a sixth sample scaled feature vector;

[0297] Performing a linear calculation on the sixth sample scaled feature vector based on a linear layer to obtain a first sample linear feature vector;

[0298] Based on two normalization layers, the first sample linear feature vector is normalized respectively to obtain a first sample normalized feature vector and a second sample normalized feature vector;

[0299] Based on the self-attention operation layer, a self-attention operation is performed on the first sample linear feature vector, the first sample normalized feature vector, and the second sample normalized feature vector to obtain a second sample self-attention feature vector;

[0300] Based on the feature scaling layer, the second sample self-attention feature vector is scaled by the first scaling factor to obtain a seventh sample scaled feature vector;

[0301] Performing a linear calculation on the seventh sample scaled feature vector based on a linear layer to obtain a second sample linear feature vector;

[0302] Activating the second sample linear feature vector based on the activation layer to obtain an activated second sample linear feature vector;

[0303] Perform feature mapping on the activated second sample linear feature vector based on the linear layer to obtain a third sample mapping feature vector;

[0304] Based on the feature scaling layer, the third sample mapping feature vector is scaled using a third scaling factor in the second noise feature vector to obtain an eighth sample scaling feature vector;

[0305] Based on the feature scaling layer, a residual calculation is performed on the eighth sample scaled feature vector and the second sample attention embedded feature vector to obtain a noise distribution in the second sample attention embedded feature vector.

[0306] Among them, the second noise feature vector can be a pre-set constant vector used to characterize the noise level in the second sample image feature matrix after adding noise. The second noise feature vector, like the above-mentioned second sample text feature vector and the second sample image feature matrix after adding noise, are inputs to the target part reconstruction model to be trained.

[0307] The sixth sample scaled feature vector may be a feature vector obtained by scaling the second sample attention embedding feature vector based on the feature scaling layer using the first scaling factor and the second scaling factor in the second noise feature vector.

[0308] The first sample linear feature vector may be a feature vector obtained by performing a linear calculation on the sixth sample scaled feature vector based on a linear layer.

[0309] The first sample normalized feature vector and the second sample normalized feature vector may be feature vectors obtained by normalizing the first sample linear feature vector based on two normalization layers.

[0310] The second sample self-attention feature vector may be a feature vector obtained after a self-attention operation is performed on the first sample linear feature vector, the first sample normalized feature vector, and the second sample normalized feature vector based on the self-attention operation layer.

[0311] The seventh sample scaled feature vector may be a feature vector obtained by scaling the second sample self-attention feature vector using the first scaling factor based on the feature scaling layer.

[0312] The second sample linear feature vector may be a feature vector obtained by performing a linear calculation on the seventh sample scaled feature vector based on a linear layer.

[0313] The third sample mapping feature vector may be a feature vector obtained by performing feature mapping on the activated second sample linear feature vector based on a linear layer.

[0314] The eighth sample scaling feature vector may be a feature vector obtained by scaling the third sample mapping feature vector using the third scaling factor in the second noise feature vector based on the feature scaling layer.

[0315] Continue to refer Fig.11 , the first scaling factor α in the second noise feature vector can be used based on the feature scaling layer x and a second scaling factor β x The second sample attention embedding feature vector is scaled to obtain the sixth sample scaled feature vector; the sixth sample scaled feature vector is linearly calculated based on the linear layer to obtain the first sample linear feature vector; the first sample linear feature vector is normalized based on two normalization layers to obtain the first sample normalized feature vector and the second sample normalized feature vector; the first sample linear feature vector, the first sample normalized feature vector and the second sample normalized feature vector are then used as QKV of the self-attention operation layer for self-attention operation to obtain the second sample self-attention feature vector; based on the feature scaling layer, the first scaling factor α is used to obtain the second sample self-attention feature vector. xThe second sample self-attention feature vector is scaled to obtain the seventh sample scaled feature vector; based on the linear layer, the seventh sample scaled feature vector is linearly calculated to obtain the second sample linear feature vector; based on the activation layer, the second sample linear feature vector is activated to obtain the activated second sample linear feature vector; based on the linear layer, the activated second sample linear feature vector is feature mapped to obtain the third sample mapping feature vector; based on the feature scaling layer, the third scaling factor γ in the second noise feature vector is used to obtain the third sample mapping feature vector. x The third sample mapping feature vector is scaled to obtain an eighth sample scaled feature vector; based on the feature scaling layer, a residual calculation is performed on the eighth sample scaled feature vector and the second sample attention embedding feature vector to obtain a noise distribution in the second sample attention embedding feature vector.

[0316] In an embodiment of the present application, in the target part reconstruction model to be trained, the noise distribution in the second sample attention embedded feature vector can be accurately obtained through the feature scaling layer, self-attention operation layer, linear layer, two normalization layers and activation layer in the noise prediction module.

[0317] In some embodiments of the present application, Fig.10 and Fig.11 As shown, in the target part reconstruction model to be trained and the diffusion model to be trained, the noise prediction module may also include: a normalization layer.

[0318] Before scaling the first sample attention embedding feature vector based on the feature scaling layer using the first scaling factor and the second scaling factor in the first noise feature vector to obtain the first sample scaled feature vector, the method involved may further include:

[0319] Normalizing the first sample attention embedding feature vector based on the normalization layer to obtain a third sample normalized feature vector;

[0320] The step of scaling the first sample attention embedding feature vector by using the first scaling factor and the second scaling factor in the first noise feature vector based on the feature scaling layer to obtain the first sample scaled feature vector may include:

[0321] The third sample normalized feature vector is scaled based on the feature scaling layer using the first scaling factor and the second scaling factor in the first noise feature vector to obtain a first sample scaled feature vector.

[0322] Among them, the third sample normalized feature vector can be a feature vector obtained after normalizing the first sample attention embedding feature vector based on the normalization layer.

[0323] Continue to refer to the above Fig.10Before scaling the first sample attention embedding feature vector by using the first scaling factor and the second scaling factor in the first noise feature vector based on the feature scaling layer, the first sample attention embedding feature vector is first normalized by using the normalization layer to obtain a third sample normalized feature vector, and then the third sample normalized feature vector is scaled by using the first scaling factor and the second scaling factor in the first noise feature vector based on the feature scaling layer to obtain a first scaled feature vector.

[0324] Before scaling the second sample attention embedding feature vector based on the feature scaling layer using the first scaling factor and the second scaling factor in the second noise feature vector to obtain the sixth sample scaled feature vector, the method involved may further include:

[0325] Normalizing the second sample attention embedding feature vector based on the normalization layer to obtain a fourth sample normalized feature vector;

[0326] The step of scaling the second sample attention embedding feature vector based on the feature scaling layer using the first scaling factor and the second scaling factor in the second noise feature vector to obtain a sixth sample scaled feature vector may include:

[0327] The third sample normalized feature vector is scaled based on the feature scaling layer using the first scaling factor and the second scaling factor in the second noise feature vector to obtain a sixth sample scaled feature vector.

[0328] Among them, the fourth sample normalized feature vector can be a feature vector obtained after normalizing the second sample attention embedding feature vector based on the normalization layer.

[0329] Continue to refer to the above Fig.11 Before scaling the second sample attention embedding feature vector by using the first scaling factor and the second scaling factor in the second noise feature vector based on the feature scaling layer, the attention embedding feature vector is first normalized by using the normalization layer to obtain a fourth sample normalized feature vector, and then the fourth normalized feature vector is scaled by using the first scaling factor and the second scaling factor in the second noise feature vector based on the feature scaling layer to obtain a sixth scaled feature vector.

[0330] In an embodiment of the present application, before scaling the first sample attention embedding feature vector, the first sample attention embedding feature vector is first normalized using a normalization layer in a noise prediction module, thereby reducing computational power when scaling the first sample attention embedding feature vector.

[0331] In some embodiments of the present application, the noise removal module removes the noise distributed in the first sample image feature matrix after adding noise to obtain the first sample image feature matrix after removing noise, which may include:

[0332] Performing dimension transformation processing on the first sample image feature matrix after adding noise based on the noise removal module to obtain the first sample image feature matrix after dimension transformation;

[0333] Normalizing the first sample image feature matrix after dimension transformation to obtain a normalized first sample image feature matrix;

[0334] A preset calculation is performed based on the normalized first sample image feature matrix to obtain the first sample image feature matrix after noise removal.

[0335] In some embodiments of the present application, after the noise distribution is predicted by the 16-layer noise prediction module, the first sample image feature matrix after adding noise can be dimensionally transformed based on the noise removal module. Specifically, the first sample image feature matrix after adding noise can be reshaped to obtain the first sample image feature matrix after dimension transformation. Then, the first sample image feature matrix after dimension transformation is normalized. Specifically, a LayerNorm operation can be performed to obtain the normalized first sample image feature matrix. Then, a preset calculation is performed based on the normalized first sample image feature matrix to obtain the first sample image feature matrix after noise removal.

[0336] In an embodiment of the present application, a noise removal module is used to perform a dimension transformation on the first sample image feature matrix after noise is added to obtain a dimensionally transformed first sample image feature matrix, and the dimensionally transformed first sample image feature matrix is ​​normalized to obtain a normalized first sample image feature matrix. A convolution calculation is performed based on the normalized first sample image feature matrix, so that the first sample image feature matrix after noise removal can be accurately obtained.

[0337] Correspondingly, in the target part reconstruction model to be trained, the noise distributed in the second sample image feature matrix after adding noise is removed based on the noise removal module to obtain the second sample image feature matrix after removing noise, which may include:

[0338] Based on the noise removal module, a dimension transformation process is performed on the second sample image feature matrix after the noise is added, so as to obtain a dimension-transformed second sample image feature matrix;

[0339] Normalizing the second sample image feature matrix after the dimension transformation to obtain a normalized second sample image feature matrix;

[0340] A preset calculation is performed based on the normalized second sample image feature matrix to obtain the second sample image feature matrix after noise removal.

[0341] In some embodiments of the present application, in the diffusion model to be trained, performing a preset calculation based on the normalized first sample image feature matrix to obtain the first sample image feature matrix after noise removal may include:

[0342] A convolution calculation is performed on the normalized first sample image feature matrix to obtain the first sample image feature matrix after noise removal.

[0343] In some embodiments of the present application, Fig.10 As shown, in the diffusion model to be trained, a preset convolution calculation, such as Conv_out calculation, is performed on the normalized first sample image feature matrix, which can reduce the number of channels of the normalized first sample image feature matrix and obtain the first sample image feature matrix after noise removal with a dimension of [64, 64, 16].

[0344] In the embodiment of the present application, in the diffusion model to be trained, different calculations may be performed on the normalized first sample image feature matrix, and the first sample image feature matrix after noise removal may be accurately obtained.

[0345] In the target part reconstruction model to be trained, performing a preset calculation based on the normalized second sample image feature matrix to obtain the second sample image feature matrix after noise removal may include:

[0346] Multi-layer perception and linear calculation are performed on the normalized second sample image feature matrix to obtain the second sample image feature matrix after noise removal.

[0347] The multi-layer perception calculation may be a calculation performed using a multi-layer perceptron (MLP).

[0348] like Fig.11 As shown, in the target part reconstruction model to be trained, multi-layer perception MLP calculation and linear calculation are performed on the normalized second sample image feature matrix, which can reduce the number of channels of the normalized second sample image feature matrix and obtain the second sample image feature matrix after noise removal with a dimension of [64, 64, 16].

[0349] In an embodiment of the present application, in the target part reconstruction model to be trained, different calculations may be performed on the normalized second sample image feature matrix, and the second sample image feature matrix after noise removal may be accurately obtained.

[0350] After using the above method to train the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder to be trained, the second text encoder to be trained, the second image encoder to be trained, the target part reconstruction model to be trained, and the second image decoder to be trained, the first image and the second image can be generated respectively based on the trained first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder, the second text encoder to be trained, the second image encoder, the target part reconstruction model to be trained, and the second image decoder to be trained.

[0351] Since the internal processing process of the text encoder, image encoder, diffusion model and image decoder for generating the first image is generally consistent with the content processing process of the text encoder, image encoder, target part reconstruction model and image decoder for generating the second image, the content processing process of the text encoder, image encoder, target part reconstruction model and image decoder for generating the second image is taken as an example for explanation below. The internal processing process of the text encoder, image encoder, diffusion model and image decoder for generating the first image is consistent with the training process of the above-mentioned first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained and the first image decoder to be trained, and will not be repeated here.

[0352] It should be noted that the content processing process of generating the text encoder, image encoder, target part reconstruction model and image decoder for the second image is consistent with the training process of the second text encoder to be trained, the second image encoder to be trained, the target part reconstruction model to be trained and the second image decoder to be trained. Therefore, the specific internal processing process of the text encoder, image encoder, target part reconstruction model and image decoder for generating the second image introduced below will not be repeated.

[0353] In some embodiments of the present application, the text encoder may include a coding layer and a transformation layer; the extracting of text features in the target part description information based on the text encoder to obtain a text feature vector corresponding to the target part description information may include:

[0354] Performing word segmentation processing on the target part description information based on the coding layer to obtain at least one target word segmentation;

[0355] Determine, based on the encoding layer, a target word segmentation vector corresponding to at least one target word segmentation according to at least one target word segmentation and a pre-set correspondence relationship between each word segmentation and the word segmentation vector corresponding to each word segmentation;

[0356] Determine a text representation vector corresponding to the target part description information based on the target word segmentation vectors corresponding to at least one target word segmentation;

[0357] Based on the transformation layer, multi-layer self-attention calculation is performed on the text representation vector to obtain the text feature vector corresponding to the description information of the target part.

[0358] The target segmentation may be the segmentation obtained after segmentation processing of the target part description information.

[0359] The target word segmentation vector may be a word segmentation vector corresponding to the target word segmentation.

[0360] The text representation vector may be a vector corresponding to the target part description information obtained based on the target word segmentation vectors corresponding to at least one target word segmentation. Specifically, it may be a vector obtained by sorting the target word segmentation vectors corresponding to at least one target word segmentation in a certain order.

[0361] In an embodiment of the present application, the target part description information can be encoded through the encoding layer in the text encoder to obtain its corresponding text representation vector, and then multi-layer self-attention calculation can be performed on the text representation vector based on the transformation layer, so that the text feature vector corresponding to the target part description information can be accurately obtained.

[0362] In some embodiments of the present application, the image encoder may include a convolution module and a transformation module; the processing of the random image based on the image encoder to generate an image feature matrix corresponding to the random image may include:

[0363] Performing image feature extraction on the random image based on the convolution module to obtain a first feature matrix corresponding to the random image;

[0364] Obtaining the dependency relationship between the elements in the first feature matrix based on the transformation module to obtain a second feature matrix;

[0365] Perform feature extraction on the second feature matrix based on the convolution module to obtain a third feature matrix;

[0366] Obtaining the dependency relationship between the elements in the third characteristic matrix based on the transformation module to obtain a fourth characteristic matrix;

[0367] Based on the convolution module, features are extracted from the fourth feature matrix to generate an image feature matrix corresponding to the random image.

[0368] The first feature matrix may be obtained by extracting image features from a random image based on a convolution module to obtain a feature matrix corresponding to the random image.

[0369] The second feature matrix can be a feature matrix obtained after the transformation module obtains the dependency relationship between the elements in the first feature matrix.

[0370] The third feature matrix may be a feature matrix obtained after the convolution module further extracts features from the second feature matrix.

[0371] The fourth characteristic matrix may be a characteristic matrix obtained after the transformation module obtains the dependency relationship between the elements in the third characteristic matrix.

[0372] In the embodiment of the present application, the random image can be processed by the convolution module and the transformation module in the image encoder to accurately generate an image feature matrix corresponding to the random image.

[0373] In some embodiments of the present application, the image decoder may include a deconvolution module and an inverse transformation module; the decoding of the image feature matrix after the noise is removed based on the image decoder to obtain the second image may include:

[0374] Based on the deconvolution module, feature decoding is performed on the image feature matrix after noise removal to obtain a fourth feature matrix;

[0375] Obtain the dependency relationship between the elements in the fourth characteristic matrix based on the inverse transformation module to obtain the third characteristic matrix;

[0376] Perform feature extraction on the third feature matrix based on the deconvolution module to obtain a second feature matrix;

[0377] Obtain the dependency relationship between the elements in the second feature matrix based on the inverse transformation module to obtain the first feature matrix;

[0378] Based on the deconvolution module, features are extracted from the first feature matrix to generate a second image.

[0379] In the embodiment of the present application, the image feature matrix after noise removal is decoded by the deconvolution module and the inverse transformation module in the image decoder, so that the second image can be accurately obtained.

[0380] In some embodiments of the present application, the target part reconstruction model may include an input processing module, a noise prediction module and a noise removal module; the target part reconstruction model is based on the text feature vector to perform denoising on the sample image feature matrix after adding noise to obtain the sample image feature matrix after removing noise, which may include:

[0381] Based on the input processing module, the text feature vector and the image feature matrix after adding noise are preprocessed to obtain the attention embedding feature vector;

[0382] Predict the noise distribution in the attention embedding feature vector based on the noise prediction module;

[0383] Based on the noise removal module, the noise distributed in the image feature matrix after the noise is added is removed to obtain the image feature matrix after the noise is removed.

[0384] Among them, the attention embedding feature vector can be a feature vector obtained after preprocessing the text feature vector and the image feature matrix after adding noise based on the input processing module.

[0385] In an embodiment of the present application, the text feature vector and the image feature matrix after adding noise are preprocessed by an input processing module to obtain an attention embedding feature vector, and then the noise distribution in the attention embedding feature vector is predicted based on a noise prediction module. Finally, the noise distributed in the image feature matrix after adding noise is removed based on a noise removal module, so that the image feature matrix after removing noise can be accurately obtained.

[0386] In some embodiments of the present application, the input processing module may include: a linear layer, a block layer, and a splicing layer; the input processing module preprocesses the text feature vector and the sample image feature matrix after adding noise to obtain the attention embedding feature vector, which may include:

[0387] Perform feature mapping on the text feature vector based on a linear layer to obtain a first mapping feature vector;

[0388] Based on the block layer, a block operation is performed on the image feature matrix after adding noise to obtain at least one feature block;

[0389] Based on the linear layer, feature mapping is performed on the feature vectors corresponding to at least one feature block to obtain at least one second mapping feature vector;

[0390] The first mapping feature vector and at least one second mapping feature vector are concatenated based on the concatenation layer to obtain an attention embedding feature vector.

[0391] The first mapping feature vector may be a feature vector obtained by performing feature mapping on the text feature vector based on a linear layer.

[0392] The second mapping feature vector may be a feature vector obtained by performing feature mapping on feature vectors corresponding to at least one feature block based on a linear layer.

[0393] In an embodiment of the present application, the text feature vector and the image feature matrix after adding noise can be preprocessed through the linear layer, blocking layer, encoding layer and splicing layer in the input processing module to accurately obtain the attention embedding feature vector.

[0394] In some embodiments of the present application, the input processing module may further include an encoding layer. Before the first mapping feature vector and the at least one second mapping feature vector are concatenated based on the concatenation layer to obtain the attention embedding feature vector, the above method may further include:

[0395] Performing rotational position encoding on at least one second mapping feature vector based on the encoding layer to obtain at least one second mapping feature vector after rotational position encoding;

[0396] The first mapping feature vector and at least one second mapping feature vector are concatenated based on the concatenation layer to obtain an attention embedding feature vector, including:

[0397] The first mapping feature vector and at least one second mapping feature vector after the rotation position encoding are concatenated based on the concatenation layer to obtain an attention embedding feature vector.

[0398] In an embodiment of the present application, at least one second mapping feature vector is encoded based on a coding layer, thereby improving the training flexibility of the target part reconstruction model.

[0399] In some embodiments of the present application, the noise prediction module may include: a feature scaling layer, a self-attention operation layer, a linear layer, two normalization layers, and an activation layer; the noise prediction module-based prediction of the noise distribution in the attention embedded feature vector may include:

[0400] Scaling the attention embedding feature vector using a first scaling factor and a second scaling factor in the noise feature vector based on the feature scaling layer to obtain a sixth scaled feature vector;

[0401] Performing a linear calculation on the sixth scaled feature vector based on a linear layer to obtain a first linear feature vector;

[0402] Normalizing the first linear feature vector based on two normalization layers to obtain a first normalized feature vector and a second normalized feature vector;

[0403] Based on the self-attention operation layer, a self-attention operation is performed on the first linear feature vector, the first normalized feature vector, and the second normalized feature vector to obtain a second self-attention feature vector;

[0404] Scaling the second self-attention feature vector using the first scaling factor based on the feature scaling layer to obtain a seventh scaled feature vector;

[0405] Performing a linear calculation on the seventh scaled feature vector based on a linear layer to obtain a second linear feature vector;

[0406] Activating the second linear feature vector based on the activation layer to obtain an activated second linear feature vector;

[0407] Performing feature mapping on the activated second linear feature vector based on the linear layer to obtain a third mapping feature vector;

[0408] Based on the feature scaling layer, the third mapping feature vector is scaled using a third scaling factor in the noise feature vector to obtain an eighth scaled feature vector;

[0409] Based on the feature scaling layer, the residual calculation is performed on the eighth scaled feature vector and the attention embedding feature vector to obtain the noise distribution in the attention embedding feature vector.

[0410] The noise feature vector is a pre-set noise level in the image feature matrix after adding noise.

[0411] The sixth scaled feature vector may be a feature vector obtained by scaling the attention embedding feature vector based on the feature scaling layer using the first scaling factor and the second scaling factor in the noise feature vector.

[0412] The first linear feature vector may be a feature vector obtained by performing a linear calculation on the sixth scaled feature vector based on a linear layer.

[0413] The first normalized feature vector and the second normalized feature vector may be feature vectors obtained by normalizing the first linear feature vector based on two normalization layers.

[0414] The second self-attention feature vector may be a feature vector obtained by performing a self-attention operation on the first linear feature vector, the first normalized feature vector, and the second normalized feature vector based on the self-attention operation layer.

[0415] The seventh scaled feature vector may be a feature vector obtained by scaling the second self-attention feature vector using the first scaling factor based on the feature scaling layer.

[0416] The second linear feature vector may be a feature vector obtained by performing a linear calculation on the seventh scaled feature vector based on a linear layer.

[0417] The third mapped feature vector may be a feature vector obtained by performing feature mapping on the activated second linear feature vector based on a linear layer.

[0418] The eighth scaled feature vector may be a feature vector obtained by scaling the third mapping feature vector using the third scaling factor in the noise feature vector based on the feature scaling layer.

[0419] In an embodiment of the present application, the noise distribution in the attention embedded feature vector can be accurately obtained through the feature scaling layer, self-attention operation layer, linear layer, two normalization layers and activation layer in the noise prediction module.

[0420] In some embodiments of the present application, the noise prediction module may further include: a normalization layer, before the feature-based scaling layer scales the attention embedding feature vector using the first scaling factor and the second scaling factor in the noise feature vector to obtain the sixth scaled feature vector, the above-mentioned method may further include:

[0421] The attention embedding feature vector is normalized based on the normalization layer to obtain a third normalized feature vector;

[0422] The attention embedding feature vector is scaled based on the feature scaling layer using the first scaling factor and the second scaling factor in the noise feature vector to obtain a sixth scaled feature vector, including:

[0423] The third normalized feature vector is scaled based on the feature scaling layer using the first scaling factor and the second scaling factor in the noise feature vector to obtain a sixth scaled feature vector.

[0424] Among them, the third normalized feature vector can be a feature vector obtained after normalizing the attention embedding feature vector based on the normalization layer.

[0425] In an embodiment of the present application, before scaling the attention embedding feature vector, the attention embedding feature vector is first normalized using a normalization layer in a noise prediction module, thereby reducing computational power when scaling the attention embedding feature vector.

[0426] In some embodiments of the present application, before step 110, the above method may further include:

[0427] Obtaining a first loss function based on the predicted sample images output by the first to-be-trained text encoder, the first to-be-trained image encoder, the to-be-trained diffusion model and the first to-be-trained image decoder, and the sample images input into the first to-be-trained text editor;

[0428] Based on the second to-be-trained text encoder, the second to-be-trained image encoder, the to-be-trained target part reconstruction model and the predicted target part image output by the second to-be-trained image decoder, and the target part region image input by the second to-be-trained text encoder, a second loss function is obtained;

[0429] The target part region in the predicted sample image is replaced with the predicted target part image to obtain a predicted image;

[0430] Based on the predicted image and the sample image, a third loss function is obtained;

[0431] The sum of the first loss function, the second loss function and the third loss function is used as the target loss function of the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder to be trained, the second text encoder to be trained, the second image encoder to be trained, the target part reconstruction model to be trained and the second image decoder to be trained.

[0432] In an embodiment of the present application, a first loss function, a second loss function and a third loss function are used to train a first text encoder to be trained, a first image encoder to be trained, a diffusion model to be trained, a first image decoder to be trained, a second text encoder to be trained, a second image encoder to be trained, a target part reconstruction model to be trained and a second image decoder to be trained. In this way, the training accuracy of the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder to be trained, the second text encoder to be trained, the second image encoder to be trained, the target part reconstruction model to be trained and the second image decoder to be trained is improved, so that an accurate text encoder, image encoder, diffusion model and image decoder for generating a first image, as well as a text encoder, image encoder, target part reconstruction model and image decoder for generating a second image can be obtained.

[0433] In order to facilitate understanding of the solution of the embodiment of the present application, the embodiment of the present application also provides another implementation of the image processing method, such as Fig.12 As shown, the image processing method may include steps 1210 to 1270.

[0434] Step 1210: Receive image description information input by the user.

[0435] Step 1220: Generate a first image corresponding to the image description information based on the diffusion model according to the image description information.

[0436] Step 1230, determine whether there is a target part in the first image, if so, execute step 1240, if not, execute step 1270 to output the first image.

[0437] Step 1240: Obtain target part description information from the image description information.

[0438] Step 1250: Generate a second image of the target part based on the target part description information based on the target part reconstruction model.

[0439] Step 1260: Replace the target part in the first image with the detection frame of the target part in the first image by using the second image to obtain a third image.

[0440] Step 1270: Output the image.

[0441] In step 1270, if the target part in the first image is replaced by the detection frame of the target part in the first image in the second image to obtain a third image, the third image is output.

[0442] The embodiment of the present application designs a dual-stream network structure according to the image characteristics. The diffusion model branch is responsible for reconstructing the global image, and the target part reconstruction branch is responsible for reconstructing the local image of the target part. Finally, the generated target part image is made to fit the global image through the image contrast loss, so as to achieve the purpose of solving the target part deformity. In addition, the target part reconstruction model in the embodiment of the present application is decoupled from the diffusion model baseline, so the target part reconstruction model can be used alone to provide users with the need to change the target part action. For example, taking the above process as an example, an image of a girl with a V-shaped gesture with her right hand is obtained. At this time, the user can enter the image description information again, such as "change the V-shaped gesture of the girl's right hand to a thumbs up". At this time, after passing through the above target part reconstruction model branch, a right hand image with a thumbs up can be obtained, and then the hand image with the thumbs up is replaced with the V-shaped gesture area in the generated image according to the hand detection frame, so as to achieve the purpose of changing the character's hand action.

[0443] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0444] Fig.13 FIG. 1 is a schematic diagram showing the structure of an image processing device according to an exemplary embodiment. Fig.13 As shown, the image processing device 1300 may include:

[0445] The first receiving module 1310 is used to receive a first input from a user;

[0446] A first generating module 1320, configured to generate image description information in response to the first input;

[0447] A second generating module 1330, configured to generate a first image corresponding to the image description information based on the image description information;

[0448] A third generating module 1340 is configured to generate a second image corresponding to the target part based on the target part description information corresponding to the target part in the image description information when it is determined that the first image includes the target part;

[0449] The fourth generating module 1350 is configured to replace the target part in the first image with the second image to obtain a third image.

[0450] In an embodiment of the present application, image description information is generated in response to a first input from a user, and then based on the image description information, a first image corresponding to the image description information can be generated. When it is determined that the first image includes a target part, a second image corresponding to the target part can be separately generated based on the target part description information corresponding to the target part in the image description information. The target part in the second image generated in this way has a better effect and will not have deformity problems. Then, the target part in the first image is replaced with the second image to obtain a third image, thereby solving the problem of deformity of the target part in the character image generated based on the Vincent graph algorithm.

[0451] In some embodiments of the present application, the third generating module 1340 may include:

[0452] A first determining unit is used to extract text features in the target part description information based on a text encoder to obtain a text feature vector corresponding to the target part description information;

[0453] A second determining unit, configured to extract features from a random image based on an image encoder, and generate an image feature matrix corresponding to the random image;

[0454] A third determining unit is used to add random noise to the image feature matrix to obtain the image feature matrix after adding noise;

[0455] A fourth determination unit is used to perform denoising processing on the image feature matrix after adding noise by using the text feature vector based on the target part reconstruction model to obtain the image feature matrix after removing noise;

[0456] The fifth determining unit is used to decode the image feature matrix after the noise is removed based on the image decoder to obtain a second image.

[0457] In some embodiments of the present application, the text encoder includes a coding layer and a transformation layer; the first determining unit is specifically used to:

[0458] Performing word segmentation processing on the target part description information based on the coding layer to obtain at least one target word segmentation;

[0459] Determine, based on the encoding layer, the target word segmentation vectors corresponding to the at least one target word segmentation and the pre-set correspondence between each word segmentation and the word segmentation vectors corresponding to each word segmentation;

[0460] Determining a text representation vector corresponding to the target part description information based on the target word segmentation vectors respectively corresponding to the at least one target word segmentation;

[0461] Based on the transformation layer, multi-layer self-attention calculation is performed on the text representation vector to obtain a text feature vector corresponding to the target part description information.

[0462] In some embodiments of the present application, the image encoder includes a convolution module and a transformation module; the second determination unit is specifically used to:

[0463] Performing image feature extraction on the random image based on the convolution module to obtain a first feature matrix corresponding to the random image;

[0464] Obtaining the dependency relationship between the elements in the first feature matrix based on the transformation module to obtain a second feature matrix;

[0465] Performing feature extraction on the second feature matrix based on the convolution module to obtain a third feature matrix;

[0466] Obtaining the dependency relationship between the elements in the third characteristic matrix based on the transformation module to obtain a fourth characteristic matrix;

[0467] Based on the convolution module, feature extraction is performed on the fourth feature matrix to generate an image feature matrix corresponding to the random image.

[0468] In some embodiments of the present application, the image decoder includes a deconvolution module and an inverse transformation module; the fifth determination unit is specifically used to:

[0469] Performing feature decoding on the image feature matrix after the noise is removed based on the deconvolution module to obtain the fourth feature matrix;

[0470] Obtaining the dependency relationship between the elements in the fourth characteristic matrix based on the inverse transformation module to obtain a third characteristic matrix;

[0471] Performing feature extraction on the third feature matrix based on the deconvolution module to obtain a second feature matrix;

[0472] Obtaining the dependency relationship between the elements in the second feature matrix based on the inverse transformation module to obtain a first feature matrix;

[0473] Based on the deconvolution module, feature extraction is performed on the first feature matrix to generate a second image.

[0474] In some embodiments of the present application, the target part reconstruction model includes an input processing module, a noise prediction module and a noise removal module; the fourth determination unit includes:

[0475] A first processing subunit is used to pre-process the text feature vector and the image feature matrix after adding noise based on the input processing module to obtain an attention embedding feature vector;

[0476] A second processing subunit, configured to predict a noise distribution in the attention embedding feature vector based on the noise prediction module;

[0477] The third processing subunit is used to remove the noise distributed in the image feature matrix after adding noise based on the noise removal module to obtain the image feature matrix after removing the noise.

[0478] In some embodiments of the present application, the input processing module includes: a linear layer, a block layer and a splicing layer; the first processing subunit is specifically used for:

[0479] Performing feature mapping on the text feature vector based on the linear layer to obtain a first mapping feature vector;

[0480] Based on the block layer, a block operation is performed on the image feature matrix after adding noise to obtain at least one feature block;

[0481] Based on the linear layer, feature mapping is performed on the feature vectors corresponding to the at least one feature block to obtain at least one second mapping feature vector;

[0482] The first mapping feature vector and the at least one second mapping feature vector are concatenated based on the concatenation layer to obtain an attention embedding feature vector.

[0483] In some embodiments of the present application, the input processing module further includes an encoding layer,

[0484] The fourth determining unit includes: a fourth processing subunit, configured to, before splicing the first mapping feature vector and the at least one second mapping feature vector based on the splicing layer to obtain the attention embedding feature vector, respectively perform rotation position encoding on the at least one second mapping feature vector based on the encoding layer to obtain the at least one second mapping feature vector after the rotation position encoding;

[0485] The first processing subunit is specifically used to: splice the first mapping feature vector and at least one second mapping feature vector after the rotation position encoding based on the splicing layer to obtain an attention embedding feature vector.

[0486] In some embodiments of the present application, the noise prediction module includes: a feature scaling layer, a self-attention operation layer, a linear layer, two normalization layers and an activation layer; the second processing subunit is used to:

[0487] Based on the feature scaling layer, the attention embedding feature vector is scaled using a first scaling factor and a second scaling factor in a noise feature vector to obtain a sixth scaled feature vector, wherein the noise feature vector is a preset noise level in a feature matrix of the image after adding noise;

[0488] Performing a linear calculation on the sixth scaled feature vector based on the linear layer to obtain a first linear feature vector;

[0489] Normalizing the first linear feature vector based on the two normalization layers to obtain a first normalized feature vector and a second normalized feature vector;

[0490] Based on the self-attention operation layer, a self-attention operation is performed on the first linear feature vector, the first normalized feature vector, and the second normalized feature vector to obtain a second self-attention feature vector;

[0491] Scaling the second self-attention feature vector using the first scaling factor based on the feature scaling layer to obtain a seventh scaled feature vector;

[0492] Performing a linear calculation on the seventh scaled feature vector based on the linear layer to obtain a second linear feature vector;

[0493] activating the second linear feature vector based on the activation layer to obtain an activated second linear feature vector;

[0494] Performing feature mapping on the activated second linear feature vector based on the linear layer to obtain a third mapping feature vector;

[0495] Based on the feature scaling layer, the third mapping feature vector is scaled using a third scaling factor in the noise feature vector to obtain an eighth scaled feature vector;

[0496] Based on the feature scaling layer, residual calculation is performed on the eighth scaled feature vector and the attention embedding feature vector to obtain a noise distribution in the attention embedding feature vector.

[0497] In some embodiments of the present application, the noise prediction module further includes: a normalization layer;

[0498] The fourth determining unit includes: a fifth processing subunit, configured to normalize the attention embedding feature vector based on the normalization layer to obtain a third normalized feature vector before scaling the attention embedding feature vector based on the feature scaling layer using the first scaling factor and the second scaling factor in the noise feature vector to obtain the first scaled feature vector;

[0499] The second processing subunit is specifically configured to: scale the third normalized feature vector using the first scaling factor and the second scaling factor in the noise feature vector based on the feature scaling layer to obtain a first scaled feature vector.

[0500] In some embodiments of the present application, the above-mentioned device may further include:

[0501] A first determination module, configured to obtain a first loss function based on the predicted sample images output by the first to-be-trained text encoder, the first to-be-trained image encoder, the to-be-trained diffusion model and the first to-be-trained image decoder, and the sample images input to the first to-be-trained text encoder;

[0502] A second determination module is used to obtain a second loss function based on the second to-be-trained text encoder, the second to-be-trained image encoder, the to-be-trained target part reconstruction model and the predicted target part image output by the second to-be-trained image decoder, and the target part area image input by the second to-be-trained text encoder;

[0503] A third determination module, used for replacing the target part area in the predicted sample image with the predicted target part image to obtain a predicted image;

[0504] A fourth determination module, configured to obtain a third loss function based on the predicted image and the sample image;

[0505] The fifth determination module is used to use the sum of the first loss function, the second loss function and the third loss function as the target loss function of the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder to be trained, the second text encoder to be trained, the second image encoder to be trained, the target part reconstruction model to be trained and the second image decoder to be trained.

[0506] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmentedreality, AR) / virtual reality (virtualreality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personalcomputer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personalcomputer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0507] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0508] The image processing device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0509] Alternatively, if Fig.14 As shown, an embodiment of the present application further provides an electronic device 1400, including a processor 1401 and a memory 1402, wherein the memory 1402 stores programs or instructions that can be executed on the processor 1401, and when the program or instructions are executed by the processor 1401, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0510] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0511] Fig.15 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0512] The electronic device 1500 includes but is not limited to: a radio frequency unit 1501, a network module 1502, an audio output unit 1503, an input unit 1504, a sensor 1505, a display unit 1506, a user input unit 1507, an interface unit 1508, a memory 1509, and a processor 1510 and other components.

[0513] Those skilled in the art will appreciate that the electronic device 1500 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1510 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.15 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0514] The user input unit 1507 is used to receive a first input from a user;

[0515] Processor 1510 is used to generate image description information in response to the first input; generate a first image corresponding to the image description information based on the image description information; when it is determined that the first image includes a target part, generate a second image corresponding to the target part based on the target part description information corresponding to the target part in the image description information; and replace the target part in the first image with the second image to obtain a third image.

[0516] In this way, by responding to the user's first input, image description information is generated, and then based on the image description information, a first image corresponding to the image description information can be generated. When it is determined that the first image includes the target part, a second image corresponding to the target part can be separately generated according to the target part description information corresponding to the target part in the image description information. The target part in the second image generated in this way has a better effect and will not have the problem of deformity. Then the target part in the first image is replaced with the second image to obtain a third image. In this way, the problem of deformity of the target part in the character image generated based on the Vincent graph algorithm is solved.

[0517] Optionally, the processor 1510 is further used to extract text features in the target part description information based on a text encoder to obtain a text feature vector corresponding to the target part description information; extract features of a random image based on an image encoder to generate an image feature matrix corresponding to the random image; add random noise to the image feature matrix to obtain an image feature matrix after adding noise; denoise the image feature matrix after adding noise using the text feature vector based on a target part reconstruction model to obtain an image feature matrix after removing noise; and decode the image feature matrix after removing noise based on an image decoder to obtain a second image.

[0518] In this way, the second image can be obtained through the text encoder, the image encoder, the target part reconstruction model and the image decoder, thereby improving the efficiency and accuracy of generating the second image corresponding to the target part description information.

[0519] Optionally, the text encoder includes an encoding layer and a transformation layer; the processor 1510 is further used to perform word segmentation processing on the target part description information based on the encoding layer to obtain at least one target word segmentation; based on the encoding layer, according to the at least one target word segmentation, and the correspondence between each pre-set word segmentation and the word segmentation vector corresponding to each word segmentation, determine the target word segmentation vector corresponding to the at least one target word segmentation; based on the target word segmentation vector corresponding to the at least one target word segmentation, determine the text representation vector corresponding to the target part description information; based on the transformation layer, perform multi-layer self-attention calculation on the text representation vector to obtain the text feature vector corresponding to the target part description information.

[0520] In this way, the target part description information can be encoded through the encoding layer in the text encoder to obtain its corresponding text representation vector, and then multi-layer self-attention calculation can be performed on the text representation vector based on the transformation layer, so that the text feature vector corresponding to the target part description information can be accurately obtained.

[0521] Optionally, the image encoder includes a convolution module and a transformation module; the processor 1510 is further used to perform image feature extraction on the random image based on the convolution module to obtain a first feature matrix corresponding to the random image; obtain the dependency relationship between each element in the first feature matrix based on the transformation module to obtain a second feature matrix; perform feature extraction on the second feature matrix based on the convolution module to obtain a third feature matrix; obtain the dependency relationship between each element in the third feature matrix based on the transformation module to obtain a fourth feature matrix; perform feature extraction on the fourth feature matrix based on the convolution module to generate an image feature matrix corresponding to the random image.

[0522] In this way, the random image can be processed through the convolution module and the transformation module in the image encoder to accurately generate the image feature matrix corresponding to the random image.

[0523] Optionally, the image decoder includes a deconvolution module and an inverse transformation module; the processor 1510 is also used to perform feature decoding on the image feature matrix after noise removal based on the deconvolution module to obtain the fourth feature matrix; obtain the dependency relationship between the elements in the fourth feature matrix based on the inverse transformation module to obtain the third feature matrix; extract features from the third feature matrix based on the deconvolution module to obtain the second feature matrix; obtain the dependency relationship between the elements in the second feature matrix based on the inverse transformation module to obtain the first feature matrix; and extract features from the first feature matrix based on the deconvolution module to generate a second image.

[0524] In this way, the image feature matrix after noise removal is decoded by the deconvolution module and the inverse transformation module in the image decoder, and the second image can be accurately obtained.

[0525] Optionally, the target part reconstruction model includes an input processing module, a noise prediction module and a noise removal module; the processor 1510 is also used to preprocess the text feature vector and the image feature matrix after adding noise based on the input processing module to obtain an attention embedding feature vector; predict the noise distribution in the attention embedding feature vector based on the noise prediction module; and remove the noise distributed in the image feature matrix after adding noise based on the noise removal module to obtain the image feature matrix after removing the noise.

[0526] In this way, the text feature vector and the image feature matrix after adding noise are preprocessed through the input processing module to obtain the attention embedding feature vector, and then the noise distribution in the attention embedding feature vector is predicted based on the noise prediction module. Finally, the noise distributed in the image feature matrix after adding noise is removed based on the noise removal module, so that the image feature matrix after removing noise can be accurately obtained.

[0527] Optionally, the input processing module includes: a linear layer, a blocking layer and a splicing layer; the processor 1510 is also used to perform feature mapping on the text feature vector based on the linear layer to obtain a first mapping feature vector; perform a blocking operation on the image feature matrix after adding noise based on the blocking layer to obtain at least one feature block; perform feature mapping on the feature vectors corresponding to the at least one feature block based on the linear layer to obtain at least one second mapping feature vector; and splice the first mapping feature vector and the at least one second mapping feature vector based on the splicing layer to obtain an attention embedding feature vector.

[0528] In this way, the text feature vector and the image feature matrix after adding noise can be preprocessed through the linear layer, blocking layer, encoding layer and splicing layer in the input processing module to accurately obtain the attention embedding feature vector.

[0529] Optionally, the input processing module also includes an encoding layer, and the processor 1510 is further used to perform rotational position encoding on the at least one second mapping feature vector based on the encoding layer to obtain at least one second mapping feature vector after rotational position encoding; and splice the first mapping feature vector and the at least one second mapping feature vector after rotational position encoding based on the splicing layer to obtain an attention embedding feature vector.

[0530] In this way, at least one second mapping feature vector is encoded based on the encoding layer, thereby improving the training flexibility of the target part reconstruction model.

[0531] Optionally, the noise prediction module includes: a feature scaling layer, a self-attention operation layer, a linear layer, two normalization layers and an activation layer; the processor 1510 is also used to scale the attention embedding feature vector based on the feature scaling layer using the first scaling factor and the second scaling factor in the noise feature vector to obtain a sixth scaled feature vector, wherein the noise feature vector is a pre-set noise level in the image feature matrix after adding noise; based on the linear layer, a linear calculation is performed on the sixth scaled feature vector to obtain a first linear feature vector; based on the two normalization layers, the first linear feature vector is normalized respectively to obtain a first normalized feature vector and a second normalized feature vector; based on the self-attention operation layer, the first linear feature vector, the first normalized feature vector and the second normalized feature vector are calculated. A second self-attention feature vector is obtained by performing self-attention operation; the second self-attention feature vector is scaled by using the first scaling factor based on the feature scaling layer to obtain a seventh scaling feature vector; a second linear feature vector is linearly calculated based on the linear layer to obtain a second linear feature vector; the second linear feature vector is activated based on the activation layer to obtain an activated second linear feature vector; a third mapping feature vector is feature mapped based on the linear layer to obtain a third mapping feature vector; the third mapping feature vector is scaled by using the third scaling factor in the noise feature vector based on the feature scaling layer to obtain an eighth scaling feature vector; a residual calculation is performed on the eighth scaling feature vector and the attention embedding feature vector based on the feature scaling layer to obtain a noise distribution in the attention embedding feature vector.

[0532] In this way, the noise distribution in the attention embedded feature vector can be accurately obtained through the feature scaling layer, self-attention operation layer, linear layer, two normalization layers and activation layer in the noise prediction module.

[0533] Optionally, the noise prediction module also includes: a normalization layer, a processor 1510, which is also used to normalize the attention embedding feature vector based on the normalization layer to obtain a third normalized feature vector; based on the feature scaling layer, the third normalized feature vector is scaled using the first scaling factor and the second scaling factor in the noise feature vector to obtain a sixth scaled feature vector.

[0534] Thus, before scaling the attention embedding feature vector, the attention embedding feature vector is first normalized using the normalization layer in the noise prediction module, thereby reducing computational power when scaling the attention embedding feature vector.

[0535] Optionally, the processor 1510 is further used to obtain a first loss function based on the predicted sample image output by the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained and the first image decoder to be trained, and the sample image input to the first text encoder to be trained; obtain a second loss function based on the predicted target part image output by the second text encoder to be trained, the second image encoder to be trained, the target part reconstruction model to be trained and the second image decoder to be trained, and the target part area image input to the second text encoder to be trained; replace the target part area in the predicted sample image with the predicted target part image to obtain a predicted image; obtain a third loss function based on the predicted image and the sample image; and use the sum of the first loss function, the second loss function and the third loss function as the target loss function of the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder, the second text encoder to be trained, the second image encoder to be trained, the target part reconstruction model to be trained and the second image decoder to be trained.

[0536] In this way, the first loss function, the second loss function and the third loss function are used to train the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder to be trained, the second text encoder to be trained, the second image encoder to be trained, the target part reconstruction model to be trained and the second image decoder to be trained, thereby improving the training accuracy of the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder to be trained, the second text encoder to be trained, the second image encoder, the target part reconstruction model to be trained and the second image decoder to be trained, thereby obtaining accurate text encoders, image encoders, diffusion models and image decoders for generating the first image, as well as text encoders, image encoders, target part reconstruction models and image decoders for generating the second image.

[0537] It should be understood that in the embodiment of the present application, the input unit 1504 may include a graphics processor (Graphics Processing Unit, GPU) 15041 and a microphone 15042, and the graphics processor 15041 processes the image data of the static picture or video obtained by the image capture device (such as a color camera) in the video capture mode or the image capture mode. The display unit 1506 may include a display panel 15061, and the display panel 15061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1507 includes a touch panel 15071 and at least one of other input devices 15072. The touch panel 15071 is also called a touch screen. The touch panel 15071 may include two parts: a touch detection device and a touch controller. Other input devices 15072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0538] The memory 1509 can be used to store software programs and various data. The memory 1509 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1509 may include a volatile memory or a non-volatile memory, or the memory 1509 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1509 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0539] The processor 1510 may include one or more processing units; optionally, the processor 1510 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1510.

[0540] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0541] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0542] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0543] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0544] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0545] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0546] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0547] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: receiving a first input from a user; In response to the first input, generating image description information; Based on the image description information, generating a first image corresponding to the image description information; In a case where it is determined that the first image includes a target part, generating a second image corresponding to the target part based on the target part description information corresponding to the target part in the image description information; The target part in the first image is replaced by the second image to obtain a third image.

2. The method according to claim 1, characterized in that The step of generating a second image corresponding to the target part based on the target part description information corresponding to the target part in the image description information includes: Extracting text features in the target part description information based on a text encoder to obtain a text feature vector corresponding to the target part description information; Extract features from a random image based on an image encoder to generate an image feature matrix corresponding to the random image; Adding random noise to the image feature matrix to obtain an image feature matrix after adding noise; Based on the target part reconstruction model, the text feature vector is used to perform denoising on the image feature matrix after adding noise, so as to obtain the image feature matrix after removing noise; The image feature matrix after the noise is removed is decoded based on an image decoder to obtain a second image.

3. The method according to claim 2, characterized in that The text encoder includes a coding layer and a transformation layer; the text feature in the target part description information is extracted based on the text encoder to obtain a text feature vector corresponding to the target part description information, including: Performing word segmentation processing on the target part description information based on the coding layer to obtain at least one target word segmentation; Determine, based on the encoding layer, the target word segmentation vectors corresponding to the at least one target word segmentation and the pre-set correspondence between each word segmentation and the word segmentation vectors corresponding to each word segmentation; Determining a text representation vector corresponding to the target part description information based on the target word segmentation vectors respectively corresponding to the at least one target word segmentation; Based on the transformation layer, multi-layer self-attention calculation is performed on the text representation vector to obtain a text feature vector corresponding to the target part description information.

4. The method according to claim 2, characterized in that: The image encoder includes a convolution module and a transformation module; The step of processing the random image based on the image encoder to generate an image feature matrix corresponding to the random image includes: Performing image feature extraction on the random image based on the convolution module to obtain a first feature matrix corresponding to the random image; Obtaining the dependency relationship between the elements in the first feature matrix based on the transformation module to obtain a second feature matrix; Performing feature extraction on the second feature matrix based on the convolution module to obtain a third feature matrix; Obtaining the dependency relationship between the elements in the third characteristic matrix based on the transformation module to obtain a fourth characteristic matrix; Based on the convolution module, feature extraction is performed on the fourth feature matrix to generate an image feature matrix corresponding to the random image.

5. The method according to claim 4, characterized in that The image decoder includes a deconvolution module and an inverse transformation module; The step of decoding the image feature matrix after the noise is removed based on the image decoder to obtain a second image includes: Performing feature decoding on the image feature matrix after the noise is removed based on the deconvolution module to obtain the fourth feature matrix; Obtaining the dependency relationship between the elements in the fourth characteristic matrix based on the inverse transformation module to obtain a third characteristic matrix; Performing feature extraction on the third feature matrix based on the deconvolution module to obtain a second feature matrix; Obtaining the dependency relationship between the elements in the second feature matrix based on the inverse transformation module to obtain a first feature matrix; Based on the deconvolution module, feature extraction is performed on the first feature matrix to generate a second image.

6. The method according to claim 2, characterized in that The target part reconstruction model includes an input processing module, a noise prediction module and a noise removal module; The method of performing denoising on the feature matrix of the sample image after adding noise by using the text feature vector based on the target part reconstruction model to obtain the feature matrix of the sample image after removing noise comprises: Preprocessing the text feature vector and the image feature matrix after adding noise based on the input processing module to obtain an attention embedding feature vector; Predicting the noise distribution in the attention embedding feature vector based on the noise prediction module; The noise distributed in the image feature matrix after adding noise is removed based on the noise removal module to obtain the image feature matrix after removing noise.

7. The method according to claim 6, characterized in that The input processing module includes: a linear layer, a block layer and a splicing layer; The preprocessing of the text feature vector and the sample image feature matrix after adding noise based on the input processing module to obtain the attention embedding feature vector includes: Performing feature mapping on the text feature vector based on the linear layer to obtain a first mapping feature vector; Based on the block layer, a block operation is performed on the image feature matrix after adding noise to obtain at least one feature block; Based on the linear layer, feature mapping is performed on the feature vectors corresponding to the at least one feature block to obtain at least one second mapping feature vector; The first mapping feature vector and the at least one second mapping feature vector are concatenated based on the concatenation layer to obtain an attention embedding feature vector.

8. The method according to claim 7, characterized in that The input processing module also includes an encoding layer, Before concatenating the first mapping feature vector and the at least one second mapping feature vector based on the concatenation layer to obtain the attention embedding feature vector, the method further includes: Performing rotational position encoding on the at least one second mapping feature vector based on the encoding layer to obtain at least one second mapping feature vector after rotational position encoding; The step of concatenating the first mapping feature vector and the at least one second mapping feature vector based on the concatenation layer to obtain an attention embedding feature vector includes: The first mapping feature vector and at least one second mapping feature vector encoded by the rotation position are spliced ​​based on the splicing layer to obtain an attention embedding feature vector.

9. The method according to claim 6, characterized in that The noise prediction module includes: a feature scaling layer, a self-attention operation layer, a linear layer, two normalization layers and an activation layer; The predicting the noise distribution in the attention embedding feature vector based on the noise prediction module includes: Based on the feature scaling layer, the attention embedding feature vector is scaled using a first scaling factor and a second scaling factor in a noise feature vector to obtain a sixth scaled feature vector, wherein the noise feature vector is a preset noise level in a feature matrix of the image after adding noise; Performing a linear calculation on the sixth scaled feature vector based on the linear layer to obtain a first linear feature vector; Normalizing the first linear feature vector based on the two normalization layers to obtain a first normalized feature vector and a second normalized feature vector; Based on the self-attention operation layer, a self-attention operation is performed on the first linear feature vector, the first normalized feature vector, and the second normalized feature vector to obtain a second self-attention feature vector; Scaling the second self-attention feature vector using the first scaling factor based on the feature scaling layer to obtain a seventh scaled feature vector; Performing a linear calculation on the seventh scaled feature vector based on the linear layer to obtain a second linear feature vector; activating the second linear feature vector based on the activation layer to obtain an activated second linear feature vector; Performing feature mapping on the activated second linear feature vector based on the linear layer to obtain a third mapping feature vector; Based on the feature scaling layer, the third mapping feature vector is scaled using a third scaling factor in the noise feature vector to obtain an eighth scaled feature vector; Based on the feature scaling layer, residual calculation is performed on the eighth scaled feature vector and the attention embedding feature vector to obtain a noise distribution in the attention embedding feature vector.

10. The method according to claim 9, characterized in that The noise prediction module further includes: a normalization layer, and before scaling the attention embedding feature vector based on the feature scaling layer using the first scaling factor and the second scaling factor in the noise feature vector to obtain a sixth scaled feature vector, the method further includes: Normalizing the attention embedding feature vector based on the normalization layer to obtain a third normalized feature vector; The step of scaling the attention embedding feature vector based on the feature scaling layer using the first scaling factor and the second scaling factor in the noise feature vector to obtain a sixth scaled feature vector includes: The third normalized feature vector is scaled based on the feature scaling layer using the first scaling factor and the second scaling factor in the noise feature vector to obtain a sixth scaled feature vector.

11. The method according to claim 1, characterized in that: Before receiving the first input from the user, the method further includes: Obtaining a first loss function based on the predicted sample images output by the first to-be-trained text encoder, the first to-be-trained image encoder, the to-be-trained diffusion model and the first to-be-trained image decoder, and the sample images input to the first to-be-trained text encoder; Obtaining a second loss function based on the second to-be-trained text encoder, the second to-be-trained image encoder, the to-be-trained target part reconstruction model and the predicted target part image output by the second to-be-trained image decoder, and the target part region image input by the second to-be-trained text encoder; Replacing the target part area in the predicted sample image with the predicted target part image to obtain a predicted image; Based on the predicted image and the sample image, obtaining a third loss function; The sum of the first loss function, the second loss function and the third loss function is used as the target loss function of the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder to be trained, the second text encoder to be trained, the second image encoder to be trained, the target part reconstruction model to be trained and the second image decoder to be trained.

12. An image processing device, characterized in that: The device comprises: A first receiving module, used to receive a first input from a user; A first generating module, configured to generate image description information in response to the first input; A second generating module, configured to generate a first image corresponding to the image description information based on the image description information; a third generating module, configured to generate a second image corresponding to the target part based on the target part description information corresponding to the target part in the image description information when it is determined that the first image includes the target part; The fourth generating module is used to replace the target part in the first image with the second image to obtain a third image.

13. The device according to claim 12, characterized in that The third generation module comprises: A first determining unit is used to extract text features in the target part description information based on a text encoder to obtain a text feature vector corresponding to the target part description information; A second determining unit, configured to extract features from a random image based on an image encoder, and generate an image feature matrix corresponding to the random image; A third determining unit is used to add random noise to the image feature matrix to obtain the image feature matrix after adding noise; A fourth determination unit is used to perform denoising processing on the image feature matrix after adding noise by using the text feature vector based on the target part reconstruction model to obtain the image feature matrix after removing noise; The fifth determining unit is used to decode the image feature matrix after the noise is removed based on the image decoder to obtain a second image.

14. The device according to claim 13, characterized in that The text encoder includes a coding layer and a transformation layer; the first determining unit is specifically used for: Performing word segmentation processing on the target part description information based on the coding layer to obtain at least one target word segmentation; Determine, based on the encoding layer, the target word segmentation vectors corresponding to the at least one target word segmentation and the pre-set correspondence between each word segmentation and the word segmentation vectors corresponding to each word segmentation; Determining a text representation vector corresponding to the target part description information based on the target word segmentation vectors respectively corresponding to the at least one target word segmentation; Based on the transformation layer, multi-layer self-attention calculation is performed on the text representation vector to obtain a text feature vector corresponding to the target part description information.

15. The device according to claim 13, characterized in that The image encoder includes a convolution module and a transformation module; the second determination unit is specifically used for: Performing image feature extraction on the random image based on the convolution module to obtain a first feature matrix corresponding to the random image; Obtaining the dependency relationship between the elements in the first feature matrix based on the transformation module to obtain a second feature matrix; Performing feature extraction on the second feature matrix based on the convolution module to obtain a third feature matrix; Obtaining the dependency relationship between the elements in the third characteristic matrix based on the transformation module to obtain a fourth characteristic matrix; Based on the convolution module, feature extraction is performed on the fourth feature matrix to generate an image feature matrix corresponding to the random image.

16. The device according to claim 15, characterized in that The image decoder includes a deconvolution module and an inverse transformation module; the fifth determination unit is specifically used for: Performing feature decoding on the image feature matrix after the noise is removed based on the deconvolution module to obtain the fourth feature matrix; Obtaining the dependency relationship between the elements in the fourth characteristic matrix based on the inverse transformation module to obtain a third characteristic matrix; Performing feature extraction on the third feature matrix based on the deconvolution module to obtain a second feature matrix; Obtaining the dependency relationship between the elements in the second feature matrix based on the inverse transformation module to obtain a first feature matrix; Based on the deconvolution module, feature extraction is performed on the first feature matrix to generate a second image.

17. The device according to claim 13, characterized in that The target part reconstruction model includes an input processing module, a noise prediction module and a noise removal module; the fourth determination unit includes: A first processing subunit is used to pre-process the text feature vector and the image feature matrix after adding noise based on the input processing module to obtain an attention embedding feature vector; A second processing subunit, configured to predict a noise distribution in the attention embedding feature vector based on the noise prediction module; The third processing subunit is used to remove the noise distributed in the image feature matrix after adding noise based on the noise removal module to obtain the image feature matrix after removing the noise.

18. The device according to claim 12, characterized in that The device also includes: A first determination module, configured to obtain a first loss function based on the predicted sample images output by the first to-be-trained text encoder, the first to-be-trained image encoder, the to-be-trained diffusion model and the first to-be-trained image decoder, and the sample images input to the first to-be-trained text encoder; A second determination module is used to obtain a second loss function based on the second to-be-trained text encoder, the second to-be-trained image encoder, the to-be-trained target part reconstruction model and the predicted target part image output by the second to-be-trained image decoder, and the target part area image input by the second to-be-trained text encoder; A third determination module, used for replacing the target part area in the predicted sample image with the predicted target part image to obtain a predicted image; A fourth determination module, configured to obtain a third loss function based on the predicted image and the sample image; A fifth determination module is used to use the sum of the first loss function, the second loss function and the third loss function as the target loss function of the first text encoder to be trained, the first image encoder to be trained, the diffusion model to be trained, the first image decoder to be trained, the second text encoder to be trained, the second image encoder to be trained, the target part reconstruction model to be trained and the second image decoder to be trained.

19. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the image processing method of the electronic device as described in any one of claims 1 to 11 are implemented.

20. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 11 are implemented.

Images