Training method for decorative element replacement model and method for replacing decorative elements of characters

Abstract

The present application relates to a training method for a decorative element replacement model, a character decorative element replacement, an apparatus, a computer device, a storage medium and a computer program product. The model training method includes: obtaining a pre-trained model for realizing decorative element image replacement; constructing incremental training samples based on complete character portraits collected at different angles for the same sample character; using the incremental training samples to incrementally train the pre-trained model so that the pre-trained model replaces the second element image of the character decorative element in the complete character portrait at a second angle based on the first element image of the character decorative element in the complete character portrait at a first angle; when the incremental training is completed using multiple incremental training samples, a decorative element replacement model with angle compensation capability is obtained. The image display effect of the character decorative element changes with the change of the character portrait angle. The above method can improve work efficiency.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method for a decorative element replacement model, a method and apparatus for replacing a character decorative element, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Artificial Intelligence (AI) is a comprehensive technology in computer science. By studying the design principles and implementation methods of various intelligent machines, machines can have the functions of perception, reasoning and decision-making. Applying artificial intelligence technology to the field of image processing, a decorative element replacement model for replacing character decorative elements can be obtained through model training.

[0003] In the traditional technology, the image processing capability of the diffusion model is used to introduce the text description of the character decoration elements into the text condition to achieve the goal of replacing the character decoration elements. In the traditional technology, during the training process, the text description needs to be constantly adjusted to obtain the desired character decoration elements and achieve the purpose of convergence of the loss function. Therefore, the traditional technology has the problem of low work efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a training method for a decorative element replacement model, a character decorative element replacement method, device, computer equipment, computer-readable storage medium and computer program product that can improve work efficiency in response to the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a method for training a decorative element replacement model. The method comprises:

[0006] A pre-trained model for implementing the replacement of decorative element images is obtained; each pre-trained sample of the pre-trained model includes a complete head portrait of a person with a decorative element of the person; and the image display effect of the decorative element of the person changes with the change of the angle of the head portrait of the person;

[0007] Construct incremental training samples based on complete head portraits collected from different angles for the same sample person;

[0008] Performing incremental training on the pre-trained model using the incremental training samples, so that the pre-trained model replaces the second element image of the character decoration element in the complete character portrait at a second angle based on the first element image of the character decoration element in the complete character portrait at a first angle;

[0009] When the pre-trained model is incrementally trained using a plurality of incremental training samples and a training end condition is met, a decorative element replacement model with angle compensation capability is obtained; the decorative element replacement model is used to generate a target image of a target person.

[0010] In a second aspect, the present application also provides a training device for a decorative element replacement model. The device comprises:

[0011] A pre-trained model acquisition module is used to acquire a pre-trained model for implementing decorative element image replacement; each pre-trained sample of the pre-trained model includes a complete character portrait with a character decorative element; the image display effect of the character decorative element changes with the change of the character portrait angle;

[0012] An incremental sample construction module, used to construct incremental training samples based on complete head portraits collected from different angles for the same sample person;

[0013] An incremental training module, configured to perform incremental training on the pre-trained model using the incremental training samples, so that the pre-trained model replaces a second element image of the character decoration element in the complete character portrait at a second angle based on a first element image of the character decoration element in the complete character portrait at a first angle;

[0014] The decorative element replacement model determination module is used to obtain a decorative element replacement model with angle compensation capability by incrementally training the pre-trained model using multiple incremental training samples and satisfying the training end conditions; the decorative element replacement model is used to generate a target image of a target person.

[0015] In a third aspect, the present application also provides a method for replacing character decoration elements. The method comprises:

[0016] Acquire a character image of a target character and a reference image including reference decorative elements;

[0017] identifying a decorative element region in the reference image to obtain a reference element image of the reference decorative element;

[0018] The reference element image and the character image are input into a decorative element replacement model to obtain a target image of the target character; the target character has the reference decorative element in the target image; the decorative element replacement model is trained based on the above-mentioned decorative element replacement model training method.

[0019] In a fourth aspect, the present application also provides a device for replacing character decoration elements. The device comprises:

[0020] An image acquisition module, used to acquire a character image of a target character and a reference image including reference decorative elements;

[0021] A decorative element recognition module, used to recognize a decorative element region in the reference image and obtain a reference element image of the reference decorative element;

[0022] The decorative element replacement module is used to input the reference element image and the character image into the decorative element replacement model to obtain the target image of the target character; the target character has the reference decorative element in the target image; the decorative element replacement model is trained based on the above-mentioned decorative element replacement model training method.

[0023] In a fifth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0024] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0025] In a seventh aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0026] The training method for the decorative element replacement model, the method, device, computer equipment, computer-readable storage medium and computer program product of the above-mentioned decorative element replacement model, based on a pre-trained model for realizing decorative element image replacement, use incremental training samples constructed based on complete character portraits collected for the same sample character at different angles to perform incremental training, so that the pre-trained model replaces the second element image of the character decorative element in the complete character portrait at a second angle based on the first element image of the character decorative element in the complete character portrait at a first angle, and when the incremental training is completed using multiple incremental training samples, a decorative element replacement model with angle compensation capability is obtained. In the above process, on the one hand, introducing the decorative element images of the character decorative elements during the training process can solve the problem of weak control ability in using text to describe the character decorative elements, avoid the need to repeatedly adjust the training samples to achieve the training goals, and is conducive to improving work efficiency; on the other hand, through two-stage training, it can ensure that the replacement ability of the character decorative elements can be learned in the pre-training stage, and the incremental training stage can solve the generalization problem caused by inconsistent angles, so that the complexity of the learning process can be reduced in the pre-training stage with a large number of training samples, and the number of training samples required can be significantly reduced in the incremental training stage with a higher learning complexity, which is conducive to reducing training costs and also can improve work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A diagram of an application environment of a training method for a decoration element replacement model and a method for replacing a character decoration element in one embodiment;

[0028] Figure 2 A flowchart of a method for training a decorative element replacement model in one embodiment;

[0029] Figure 3 A schematic diagram of a model structure of a pre-trained model in one embodiment;

[0030] Figure 4 A schematic diagram of rectangular frame coordinates of a face image area in one embodiment;

[0031] Figure 5 A schematic diagram of the structure of a BiseNet model in one embodiment;

[0032] Figure 6 is a schematic diagram of the structure of a BLIP model in one embodiment;

[0033] Figure 7 A schematic diagram of a process of obtaining a text description of a complete character portrait based on a BLIP model in one embodiment;

[0034] Figure 8A schematic diagram of a process of obtaining various training data based on a complete character portrait in one embodiment;

[0035] Fig. 9 A schematic diagram of a model structure of a pre-trained model in another embodiment;

[0036] Fig.10 A schematic diagram of input and output of a pre-trained model in one embodiment;

[0037] Fig.11 A schematic diagram of input and output of a pre-trained model in another embodiment;

[0038] Fig.12 is a schematic diagram of a feature fusion principle in an embodiment;

[0039] Fig.13 A flowchart of a method for training a decorative element replacement model in another embodiment;

[0040] Fig.14 A schematic diagram of a flow chart of a decoration element replacement method in one embodiment;

[0041] Fig.15 A schematic diagram of a hairstyle replacement effect in an embodiment;

[0042] Fig.16 A structural block diagram of a training device for a decoration element replacement model in one embodiment;

[0043] Fig.17 is a structural block diagram of a decorative element replacement device in one embodiment;

[0044] Fig.18 is an internal structure diagram of a computer device in one embodiment;

[0045] Fig.19 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] The training method of the decoration element replacement model and the character decoration element replacement method provided in the embodiments of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through a network. The communication network can be a wired network or a wireless network. Therefore, the terminal 102 and the server 104 can be directly or indirectly connected through wired or wireless communication. For example, the terminal 102 can be indirectly connected to the server 104 through a wireless access point, or the terminal 102 is directly connected to the server 104 through the Internet, and this application does not limit this.

[0048] Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The embodiment of the present application can be applied to the training of decorative element replacement models and the replacement of character decorative elements. The terminal 102 can be installed with a client related to character image processing, which can be software (such as a browser, video software, etc.), or a web page, a small program, etc. The server 104 is a background server corresponding to the software or a web page, a small program, etc., or a server specifically used for decorative element replacement model training or character decorative element replacement. In some embodiments, the training of the decorative element replacement model or the replacement of character decorative elements can also be achieved through the same server, which is not specifically limited in this application. Furthermore, the server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The data storage system can store data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers.

[0049] It should be noted that the training method of the decorative element replacement model and the character decorative element replacement method in the embodiment of the present application can be executed by the terminal 102 or the server 104 alone, or can be executed by the terminal 102 and the server 104 together. Taking the case where the server 104 is executed alone as an example, in the process of training the decorative element replacement model, the server 104: obtains a pre-trained model for realizing decorative element image replacement; constructs incremental training samples based on complete character portraits collected for the same sample character at different angles; uses the incremental training samples to incrementally train the pre-trained model so that the pre-trained model replaces the second element image of the character decorative element in the complete character portrait at a second angle based on the first element image of the character decorative element in the complete character portrait at a first angle; when the pre-trained model is incrementally trained using multiple incremental training samples and the training end condition is met, a decorative element replacement model with angle compensation capability is obtained. Among them, each pre-trained sample of the pre-trained model includes a complete character portrait with character decorative elements; the image display effect of the character decorative elements changes with the change of the character portrait angle; the decorative element replacement model is used to generate a target image of the target character.

[0050] The trained model can be used to replace the decorative elements of characters. Specifically, in the process of replacing the decorative elements of characters, the server 104: obtains the character image of the target character and the reference image containing the reference decorative elements; identifies the decorative element area in the reference image to obtain the reference element image of the reference decorative element; inputs the reference element image and the character image into the decorative element replacement model to obtain the target image of the target character; the target character has the reference decorative element in the target image. Among them, the character head portrait angles corresponding to the target character and the reference decorative element can be the same or different. The reference decorative element in the target image corresponds to the same character head portrait angle as the target character. That is to say, in the case where the target character and the reference decorative element each correspond to different character head portrait angles, since the decorative element replacement model has the angle compensation capability, it can generate a target image in which the reference decorative element corresponds to the same character head portrait angle as the target character.

[0051] In one embodiment, Figure 2 As shown, a training method for a decoration element replacement model is provided, which can be executed by a computer device, which can be Figure 1 The terminal or server shown in this embodiment is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0052] Step S202, obtaining a pre-trained model for implementing decorative element image replacement.

[0053] The pre-trained model refers to a machine learning model that has the ability to replace character decorative elements according to decorative element images through pre-training. The specific network structure of the pre-trained model is not unique, and may include at least one of CNN (Convolutional Neural Network), GAN (Generative Adversarial Network) or SD (Stable Diffusion). In an optional embodiment, the pre-trained model uses the Stable Diffusion model as the base model to realize the Wenshengtu function. Figure 3 As shown, the server can build a reference network with the same structure as the target network on the basis of the target network based on Stable Diffusion. The reference network is used to encode the input decorative element image to obtain the decorative element features, while the target network is used to obtain the non-decorative features of other areas in the complete character portrait except the decorative element area. By sharing attention, the decorative element features and non-decorative features are integrated to achieve the purpose of replacing the character decorative elements in the complete character portrait according to the decorative element image. The training goal of the pre-training stage is to make the model output image after replacing the character decorative elements close to the complete character portrait. That is, in the pre-training stage, the complete character portrait in the pre-training sample, on the one hand, serves as a source of image information of decorative element features and non-decorative features, and on the other hand, serves as a comparison of the model prediction results. When the pre-training is completed using a pre-training sample set containing multiple pre-training samples, a pre-training model with the ability to replace the character decorative elements according to the decorative element image is obtained.

[0054] Each pre-trained sample of the pre-trained model includes a complete character portrait with character decoration elements. The decoration element image refers to the image of the character decoration element. The character decoration element refers to the element used to decorate the character portrait. The character decoration element may include hairstyle, hat, glasses, earrings, etc. Further, the image display effect of the character decoration element changes with the change of the character portrait angle. Exemplarily, the display effects of the character decoration elements such as hairstyle, hat, glasses or earrings in each complete character portrait obtained by collecting the same character at different angles are different. In practical applications, character decoration elements play an important role in art and design. Through different decorative means and styles, the expressiveness and visual impact of the character image can be enhanced. Based on this, the application scenarios of character decoration element replacement are very wide. In the field of character design, based on the character decoration element replacement function, users can replace character decoration elements according to their needs to achieve personalized character customization; in the field of fashion design, designers can use the character decoration element replacement function to make partial adjustments to the character portrait, select appropriate character decoration elements, and improve efficiency; in virtual reality and augmented reality applications, the character decoration element replacement function can enhance the realism and immersion of user experience.

[0055] Specifically, the server can use a pre-training sample set containing multiple pre-training samples to perform model training on the initial model in advance, and obtain a pre-training model for realizing the replacement of decorative element images. It can be understood that since the decorative element image in the pre-training sample is part of the complete character portrait, the two correspond to the same character portrait angle. Therefore, the pre-training model obtained in the pre-training stage does not have the ability to compensate for angles. In actual applications, there may be an angle difference between the decorative element image as the replacement target and the complete character portrait as the replacement object. If the pre-training model is used directly to replace the decorative elements, the display effect of the character decorative element image may be distorted due to the existence of angle differences. Based on this, the present application also performs incremental training based on the pre-training model.

[0056] There is no unique way to separate the decorative element image from the complete character portrait. Optionally, the server can first identify the character decorative elements in the complete character portrait, and then extract the border of the character decorative elements, and perform image segmentation on the complete character portrait based on the border to obtain the decorative element image; Optionally, the server can also use tools such as OpenCV and dlib to perform face detection and key point extraction to determine the unique location of the character decorative elements in the complete character portrait, and then separate the task costume elements from the image through grayscale and thresholding to obtain the decorative element image; Optionally, the server can also use artificial intelligence algorithms to perform high-precision image segmentation, such as semantic segmentation and instance segmentation, to identify the character decorative elements in the complete character portrait, and further extract the decorative element image of the character decorative elements.

[0057] In one embodiment, the process of obtaining a complete character portrait includes: obtaining multiple candidate character images; performing character recognition on each candidate character image to determine the face size of the character contained in each candidate character image; for each candidate character image, if the face size of the character in the candidate character image meets the size condition and the character contains character decoration elements, obtaining a complete character portrait with character decoration elements based on the candidate character image.

[0058] Among them, the candidate character image refers to an image containing a person. The number of people contained in the candidate character image may be one or more. The face size can be represented by the proportion of the face in the entire candidate character image, or by the number of pixels in the image area where the face is located. It can be understood that the lower the proportion of the face in the entire picture (i.e., the candidate character image), the fewer the number of pixels in the face image area, and the greater the possibility of blurring the person. Based on this, the size condition may refer to the number of pixels in the face image area being greater than or equal to the pixel amount threshold, or the number of pixels in the face in the entire picture being greater than or equal to the ratio threshold. The pixel threshold may be, for example, 30,000 or 40,000, and the ratio threshold may be, for example, 1 / 16 or 1 / 15. In an optional embodiment, in order to ensure the quality of the face image, it is proposed to retain only pictures with a face proportion of more than 1 / 16. Taking a 1024*1024 picture as an example, a square face with a proportion of 1 / 16 contains 256*256 pixels, which is sufficient to represent the details of the person.

[0059] Specifically, the server can obtain multiple candidate character images from an open source image data set. Then, character recognition is performed on each candidate character image to determine the face size of the character contained in each candidate character image. In the case where the face size of the character in the candidate character image meets the size condition and the character contains character decoration elements, the candidate character image is retained, and a complete character portrait with character decoration elements is obtained based on the candidate character image. Optionally, in the case where the retained candidate character image is a single-person image, the server can determine the candidate character image as a complete character portrait with character decoration elements; in the case where the retained candidate character image is a multi-person image, the server can separate the single-person image whose face size meets the size condition and contains character decoration elements from the candidate character image to obtain a complete character portrait with character decoration elements.

[0060] In an optional embodiment, the server may use the RetinaFace model (retinal face detection model) to achieve person recognition. The RetinaFace model is a neural network model for face detection and face alignment. Its principle is based on a one-stage target detector, using multi-scale feature fusion and multi-task learning methods to accurately detect and align faces at different scales and directions. The RetinaFace model has good robustness in multi-scale, multi-direction and occlusion situations, and is suitable for real-time face detection and related applications. Figure 4 As shown in the figure, the candidate character image is input into the RetinaFace model, and the rectangular frame coordinates (x, y, w, h) of the face image area are returned. Among them, (x, y) is the coordinate of the upper left corner of the rectangle, and w and h are the width and height of the rectangle respectively. After being processed by the RetinaFace model, the number of characters contained in the candidate character image and the face size of each character can be obtained.

[0061] In an optional embodiment, before performing character recognition, a round of image screening can be performed to further ensure image quality and improve work efficiency. Specifically, the server can screen images based on aesthetic quality scores. Aesthetic quality score is an image quality assessment indicator used in the Laion-5B dataset. The score range is [0,10]. The higher the score, the better the quality of the image. Referring to the criteria for screening the Laion-5B dataset, character recognition can be performed only on candidate character images with a score greater than 6.5.

[0062] In an optional embodiment, the number of pixels on the short side of the final complete portrait of the person is not less than 1024 to further ensure clarity.

[0063] In the above embodiment, based on the candidate character images whose face sizes meet the size conditions, a complete character portrait with character decoration elements is obtained, which can ensure the image quality of the complete character portrait and further ensure the training effect.

[0064] Step S204, constructing incremental training samples based on complete head portraits collected from different angles for the same sample person.

[0065] Among them, the complete character portraits collected for the same sample character at different angles include at least a first complete character portrait collected for the sample character at a first angle, and a second complete character portrait collected for the sample character at a second angle. The first angle and the second angle refer to different character portrait angles, and the terms "first" and "second" used are only used to distinguish the first character portrait angle from another character portrait angle. Exemplarily, the first angle and the second angle may refer to any two of a front angle, a three-quarter side angle, an upward angle, a downward angle, or a test angle.

[0066] In the pre-training stage, the character decorative elements in the decorative element image correspond to the same character portrait angle as the character in the complete character portrait. The training goal of the incremental training stage is to improve the angle compensation ability of the model. Based on this, the server can obtain complete character portraits collected from different angles for the same sample character. These complete character portraits have the same character decorative elements. Among them, the same character decorative elements can be the same hairstyle, the same hat, the same pair of glasses, etc. After obtaining multiple complete character portraits collected for the same sample task, the server can construct incremental training samples based on these multi-angle complete character portraits. Optionally, the server can use the complete character portraits collected from two different angles for the same sample character as incremental training samples. Optionally, when more than two complete character portraits collected from different angles for the same sample character are obtained, the server can combine them in pairs to obtain multiple incremental training samples; the server can also perform further screening according to the image quality, select any two complete character portraits, and construct incremental training samples. Among them, the specific method of further screening according to the image quality refers to the method of screening complete character portraits from multiple candidate character images above, which is not limited here.

[0067] In an optional embodiment, the server may construct an incremental training sample including a decorative element image at a first angle and a partial character portrait with the decorative element area deleted at a second angle based on complete character portraits collected at different angles for the same sample character. Specifically, the server may obtain complete character portraits collected at a first angle and a second angle, separate the decorative element image from the complete character portrait at the first angle, and mask the decorative element area where the character decorative elements are located in the complete character portrait collected at the second angle to obtain the partial character portrait with the decorative element area deleted at the second angle. The specific method of separating the decorative element image from the complete character portrait at the second angle is described above and will not be repeated here.

[0068] Step S206, incrementally train the pre-trained model using incremental training samples, so that the pre-trained model replaces the second element image of the character decoration element in the complete character portrait at the second angle based on the first element image of the character decoration element in the complete character portrait at the first angle.

[0069] The first element image and the second element image refer to the image information of the same character decoration element of the same sample character presented at different angles. For example, the first element image and the second element image may refer to the hairstyle images of the sample character presented at different angles; for another example, the first element image and the second element image may refer to the hat images of the sample character worn at different angles.

[0070] As mentioned above, in the pre-training stage, the decorative element image separated from the complete character portrait is used to replace the character decorative element in the complete character portrait. Therefore, the training goal of the pre-training stage is to make the model output image after replacing the character decorative element close to the complete character portrait. In the incremental training stage, the model structure remains unchanged, the replacement target is the decorative element image separated from the complete character portrait at the first angle, and the replacement object is the complete character portrait collected at the second angle. Therefore, the training goal of the incremental training stage is to make the model output image after replacing the character decorative element close to the complete character portrait collected at the second angle. In other words, the server can use incremental training samples to perform incremental training on the pre-trained model, so that the pre-trained model replaces the second element image of the character decorative element in the complete character portrait at the second angle based on the first element image of the character decorative element in the complete character portrait at the first angle.

[0071] Step S208, when the pre-trained model is incrementally trained using a plurality of incremental training samples and the training end condition is met, a decorative element replacement model with angle compensation capability is obtained.

[0072] Among them, the number of training samples required in the incremental training stage is much smaller than the number of training samples required in the pre-training stage.

[0073] Specifically, the server uses an incremental training sample set including multiple incremental training samples to perform incremental training on the pre-trained model, and obtains a decorative element replacement model with angle compensation capability when a training end condition is met. The training end condition may refer to that the accuracy of the validation set is no longer improved, or that the model has reached a set maximum number of iterations, or that the loss function converges based on the change of the loss function with the number of iterations.

[0074] In an optional embodiment, the pre-trained model uses the diffusion model as the base model. The objective function is:

[0075]

[0076] The diffusion model is a generative model that synthesizes the required data samples through iterative denoising. Diffusion training includes a forward diffusion process and a reverse denoising process. In the forward diffusion process, the noise is sampled and added to the data samples based on a predefined noise schedule. This process generates noise samples with a time step of t. On the contrary, in the reverse denoising process, the denoising model Pick , and optional additional conditions C (text, pictures or voice, etc.) as input, with the prediction of added noise as the prediction target. Under the above objective function, different training data sets are used to train the model. Taking the case where the character decorative element is the hairstyle as an example, in the first stage (i.e., the pre-training stage), a large number of complete character portraits containing hairstyles are used as pre-training samples. In the Inpainting (image repair) mode, the hairstyle in the complete character portrait is replaced according to the reference hairstyle image separated from the complete character portrait; in the second stage (i.e., the incremental training stage), a small number of complete character portraits of the same person with the same hairstyle at different angles are used for incremental training. According to the reference hairstyle image separated from the complete character portrait at the first angle, the hairstyle in the complete character portrait at the second angle is replaced. Using two stages for model training can significantly reduce the training cost while ensuring the generalization of the model.

[0077] Furthermore, the decorative element replacement model is used to generate a target image of a target person. Specifically, a computer device equipped with the decorative element replacement model can obtain a character image of a target person and a reference image containing a reference decorative element; identify a decorative element region in the reference image to obtain a reference element image of the reference decorative element; input the reference element image and the character image into the decorative element replacement model to obtain a target image of the target person. The target person has a reference decorative element in the target image, and the reference decorative element in the target image corresponds to the same character head angle as the target person; the character head angles corresponding to the target person and the reference decorative element can be the same or different.

[0078] The training method of the decorative element replacement model is based on a pre-trained model for realizing decorative element image replacement, and uses incremental training samples constructed based on complete character portraits collected at different angles for the same sample character to perform incremental training, so that the pre-trained model replaces the second element image of the character decorative element in the complete character portrait at a second angle based on the first element image of the character decorative element in the complete character portrait at a first angle, and obtains a decorative element replacement model with angle compensation capability when the incremental training is completed using multiple incremental training samples. In the above process, on the one hand, the decorative element image of the character decorative element is introduced in the training process, which can solve the problem of weak control capability existing in the use of text to describe the character decorative element, avoid the need to repeatedly adjust the training samples to achieve the training goal, and is conducive to improving work efficiency; on the other hand, through two stages of training, it can ensure that the replacement capability of the character decorative element can be learned in the pre-training stage, and the incremental training stage can solve the generalization problem caused by inconsistent angles, so that the complexity of the learning process can be reduced in the pre-training stage with a large number of training samples, and the number of required training samples can be significantly reduced in the incremental training stage with a high learning complexity, which is conducive to reducing training costs and also can improve work efficiency.

[0079] In one embodiment, based on complete character portraits collected at different angles for the same sample character, incremental training samples are constructed, including: obtaining a first complete character portrait collected for the sample character at a first angle, and a second complete character portrait collected for the sample character at a second angle; identifying a first element region where a character decorative element is located in the first complete character portrait, and obtaining a first mask image for masking the first element region at the first angle; fusing the first complete character portrait and the first mask image, and separating a first element image of the first element region from the first complete character portrait; and constructing incremental training samples including the second complete character portrait and the first element image.

[0080] The first mask image at the first angle is used to mask the first element region where the character decoration element is located in the first complete character portrait. That is, in the first mask image, the first element region is black, and the corresponding pixel value is 0, and the other regions except the first element region are white, and the corresponding pixel value is 255.

[0081] Specifically, the server can obtain a first complete head portrait of a sample person collected at a first angle and a second complete head portrait of the sample person collected at a second angle. Then, a first element image as a replacement target is separated from the first complete head portrait, and then further combined with the second complete head portrait as a replacement target to construct an incremental training sample in the incremental training stage.

[0082] In the process of separating the first element image from the first complete person portrait, the server can identify the first element region where the character decoration element is located in the first complete person portrait based on image semantic segmentation, or by applying image editing software or automatic mask generation tools, and generate a first mask image for masking the first element region at a first angle. After obtaining the first mask image, the server can fuse the first complete person portrait and the first mask image to separate the first element image of the first element region from the first complete person portrait.

[0083] In an optional embodiment, the server may perform pixel value transformation on the first mask image, so that the pixel value of the first element region changes from 0 to 1, and the pixel value of other regions except the first element region changes from 255 to 0, to obtain the first mask transformation image. The pixel matrices of the first complete person portrait and the first mask transformation image are dot-multiplied, so that the pixel value of the first element region in the first complete person portrait remains unchanged, and the pixel value of other regions except the first element region changes to 0, thereby realizing the separation of the first element region and obtaining the first element image.

[0084] The pre-training sample construction process is similar to the incremental training sample construction process. In a specific embodiment, the pre-training sample construction process includes: obtaining a complete character portrait with character decorative elements; identifying the decorative element region where the character decorative elements are located in the complete character portrait, and obtaining a decorative element mask map of the complete character portrait; fusing the complete character portrait and the decorative element mask map to obtain a decorative element image in the decorative element region; and constructing a pre-training sample including the complete character portrait and the decorative element image.

[0085] In the above embodiment, the first element image of the first element area is separated from the first complete character portrait as a part of the incremental training sample, which can clarify the replacement target of the decorative element and improve the training effect.

[0086] In a specific implementation, a first complete character portrait and a first mask image are fused to separate a first element image of a first element area from the first complete character portrait, including: normalizing the first mask image to obtain a first normalized mask image; and masking the first complete character portrait using the first normalized mask image to separate the first element image of the first element area from the first complete character portrait.

[0087] Specifically, the server may perform normalization processing on the first mask image to obtain a first normalized mask image in which the pixel value of the first element region is 1 and the pixel values ​​of other regions except the first element region are 0. Then, the first normalized mask image is used to perform mask processing on the first complete person portrait to separate the first element image of the first element region from the first complete person portrait.

[0088] Exemplarily, the first mask image is defined as M1, and the first complete character portrait is defined as I1. Then, the process of determining the first element image S1 presented by the character decoration element in the first complete character portrait can be expressed as:

[0089] S1=1-(M1 / 255)

[0090] Among them, "1-(M1 / 255)" normalizes M1 to the range of [0,1], the pixel values ​​of the originally masked black area (i.e., the first element area) are normalized to 1, and the pixel values ​​of other areas that were not originally masked (i.e., other areas except the first element area) are normalized to 0. After normalization, it is multiplied with I1, so that the pixel values ​​of other areas except the first element area in I1 become 0, and the pixel values ​​of the first element area remain unchanged, thereby obtaining the first element image S1 of the first element area.

[0091] In the above embodiment, the pixel value conversion is achieved through normalization processing, the algorithm is simple, the processing process can be simplified, and it is beneficial to further improve the work efficiency.

[0092] In a specific implementation, the incremental training samples also include a partial portrait of the character at a second angle; the pre-trained model specifically completes the partial portrait of the character based on the first element image. In the case of this embodiment, the training method of the decorative element replacement model also includes: identifying the second element area where the character decorative element is located in the second complete character portrait, and obtaining a second mask image for masking the second element area at a second angle; normalizing the second complete character portrait and the second mask image respectively to obtain a normalized portrait and a second normalized mask image; using the second normalized mask image to mask the normalized portrait to obtain normalized partial portrait information; denormalizing the partial portrait information to obtain a partial character portrait with the second element area deleted.

[0093] The second mask image at the second angle is used to mask the second element region where the character decoration element is located in the second complete character portrait. That is, in the second mask image, the second element region is black, and the corresponding pixel value is 0, and the other regions except the second element region are white, and the corresponding pixel value is 255.

[0094] Specifically, the server can identify the second element region where the character decoration element is located in the second complete character portrait, obtain a second mask image for masking the second element region at a second angle, and then normalize the second complete character portrait to obtain a normalized portrait, and normalize the second mask image to obtain a second normalized mask image in which the pixel value of the second element region is 0 and the pixel value of other regions except the second element region is 1. Then, the server uses the second normalized mask image to mask the normalized portrait to obtain normalized local portrait information, and denormalizes the local portrait information to obtain a local character portrait with the second element region deleted. Among them, the algorithm for denormalizing the local portrait information corresponds to the algorithm for normalizing the second complete character portrait.

[0095] Exemplarily, the second mask image is defined as M2, and the second complete person portrait is defined as I2. Then, the process of determining to delete the partial person portrait D2 of the second element region can be expressed as:

[0096]

[0097] Among them, use Normalize I2 to the range [-1,1], Normalize M2 to the range of [0,1], normalize the pixel values ​​of the originally masked black area (i.e., the second element area) to 0, and normalize the pixel values ​​of other areas that were not originally masked (i.e., areas other than the second element area) to 1. After dot multiplication of the two normalized results, the pixel values ​​of the second element area in I2 become 0, and the pixel values ​​of other areas other than the second element area remain unchanged. After the inverse normalization operation, I2 can be restored to the range of [0,255]. After the above operation, the second element area in the second complete portrait I2 becomes gray, and other areas remain unchanged, achieving the purpose of deleting the second element area.

[0098] In the above embodiment, on the one hand, normalization is performed before masking, which can simplify the amount of calculation in the masking process and is beneficial to improving work efficiency; on the other hand, by combining normalization and denormalization, the deleted second element area is made gray instead of black, which can provide a smoother transition effect, enhance the visual hierarchy, improve editing flexibility, and reduce visual interference, which is beneficial to improving the training effect of the model.

[0099] In an optional embodiment, the server may use a BisNet model (bi-associative information network model) to segment the decorative element area where the character decorative elements are located in the complete character portrait. The complete character portrait may include the complete character portrait in the training sample during the pre-training process, and the first complete character portrait and the second complete character portrait during the incremental training process. The BiseNet model is a neural network model for image segmentation tasks, which can realize pixel-level division of different objects or areas in an image. Its structure diagram is shown in FIG. Figure 5 As shown in the figure. The BiseNet model contains two branch networks: detail branch and semantic branch. The detail branch is responsible for extracting spatial detail information, that is, low-level information. Therefore, this branch needs to be configured with rich channel capacity and set a shallow structure with a small sliding distance (Stride). In other words, the detail branch needs a large number of channels and a small number of layers. For the semantic branch, it is necessary to consider the requirements of large receptive field and small amount of computation. Specifically, the structure of the semantic branch can be designed by referring to lightweight networks such as Xception (extremely deep network), MobileNet (mobile network), and ShuffleNet (shuffle network). In contrast to the shallow characteristics of the detail branch with a large number of channels, the semantic branch requires a deep structure with a small number of channels. Because the detail branch and the semantic branch focus on different features, the detail branch extracts low-level detail features, while the semantic branch extracts high-level semantic features. On this basis, the complementary information from the two branches can be fused through a bilateral guided aggregation layer.

[0100] In an optional embodiment, the training samples of the decorative element replacement model also include text descriptions of complete character portraits. In the case of this embodiment, the training method of the decorative element replacement model also includes: for each complete character portrait, using the trained multimodal model to generate a text description of the complete character portrait.

[0101] The complete character portrait may include the complete character portrait in the training sample during the pre-training process, and the first complete character portrait and the second complete character portrait during the incremental training process. The generated text description is used to guide the model to replace the decorative element image during the pre-training or incremental training process. The model loss of the multimodal model during the training process includes image-text comparison loss, image-text matching loss and language model loss.

[0102] For example, the BLIP (Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation) model can be used to generate the description information of the complete person portrait. Figure 6 As shown in the figure, the BLIP model consists of four parts: Image Encoder, Text Encoder, Image-grounded Text Encoder, and Image-grounded Text Decoder. Image-groundedText Encoder is a binary classification model responsible for determining whether the meanings expressed by text and images are consistent. In the application scenario of this application, Figure 7 As shown, the Image-grounded Text Decoder module of BLIP can be used to generate a text description of the complete character portrait.

[0103] In one embodiment, the process of constructing training samples is described by taking the case where the character decoration element is a hairstyle as an example. Figure 8 As shown, on the one hand, the server can segment the hairstyle area from the complete person portrait I through the BiseNet model to obtain the hairstyle mask image M. And by combining normalization processing and pixel multiplication operation, the local person portrait D with the hairstyle area deleted and the hairstyle image S are further obtained. On the other hand, the server can apply the BLIP model to generate a text description of the complete person portrait I. Then, a training sample including the complete person portrait, the text description of the complete person portrait, the hairstyle mask image, the hairstyle image, and the local person portrait with the hairstyle area deleted can be obtained. As shown in FIG. Figure 3 As shown, the complete person portrait, the text description of the complete person portrait, the hairstyle mask map, and the partial person portrait with the hairstyle area deleted can be used as the input of the target network, and the hairstyle map can be used as the input of the reference network.

[0104] The following is an introduction to the model pre-training process.

[0105] In one embodiment, the training process of the pre-trained model includes: for each complete character portrait in the pre-training sample set, identifying the decorative element area in the complete character portrait, and obtaining the decorative element mask map of the complete character portrait; performing image feature extraction on the decorative element image in the complete character portrait, and obtaining the decorative element features of the decorative element image; under the guidance of the decorative element mask map, performing image feature extraction on the complete character portrait, and obtaining non-decorative features of other areas in the complete character portrait except the decorative element area; based on the character features obtained by fusing the decorative element features and the non-decorative features, decoding to obtain an updated image corresponding to the complete character portrait; when the loss statistics of each complete character portrait converge, obtaining a pre-trained model for realizing decorative element image replacement.

[0106] The model loss corresponding to any complete person portrait is used to characterize the difference between the complete person portrait and the updated image corresponding to the complete person portrait. In other words, the training goal of the pre-training process is to make the model output image after replacing the character's decorative elements close to the complete person portrait.

[0107] Specifically, the server can identify the decorative element area in the complete character portrait for each complete character portrait in the pre-training sample set based on image semantic segmentation, or by applying image editing software or automatic mask generation tools, and then obtain the decorative element mask map of the complete character portrait. When the decorative element mask map is obtained, the server can, on the one hand, separate the decorative element image of the decorative element area from the complete character portrait based on the decorative element mask map, and perform image feature extraction on the decorative element image to obtain the decorative element features. On the other hand, under the guidance of the decorative element mask map, image feature extraction can be performed on the complete character portrait to obtain non-decorative features of other areas in the complete character portrait except the decorative element area. Then, the server can obtain the complete task features by fusing the decorative element features and the non-decorative features, and decode based on the character features to obtain the updated image corresponding to the complete character portrait.

[0108] For each complete person portrait used as a training sample, the model loss during the pre-training process is used to characterize the difference between the complete person portrait and the corresponding updated image of the complete person portrait. When the loss statistics of each complete person portrait converge, the server can obtain a pre-trained model for implementing the replacement of the decorative element image. The loss statistics of each complete person portrait can be the average, weighted average or sum of the model losses of each complete person portrait, which is not limited here.

[0109] In an optional embodiment, if Figure 3As shown in the figure, the network structure of the decorative element replacement model and the pre-trained model is the same, including a target network, a reference network and a decoder. The reference network is used to extract image features from the decorative element image to obtain decorative element features; the target network is used to extract non-decorative features of other areas in the complete character portrait except the decorative element area. The decorative element features obtained by the reference network will be passed to the target network. Therefore, the target network is also used to fuse the decorative element features and non-decorative features to obtain complete character features. The character features are used as the input of the decoder so that the decoder decodes the character features to obtain an updated image corresponding to the complete character portrait, that is, the model output image.

[0110] In a specific implementation, such as Fig. 9 As shown in the figure, the pre-trained model and the decorative element replacement model use the StableDiffusion model as the base model to realize the Wenshengtu function. Specifically, in the original Stable Diffusion model, the number of input image channels is 3, and the number of channels is 4 after encoding by the Variational Autoencoder (VAE Encoder). In order to change only the decorative element area and keep the other parts of the complete character portrait unchanged, the Inpainting mode is used to add 5 additional channels, namely, a 1-channel decorative element mask map and a 4-channel partial character portrait after deleting the decorative element area. Among them, the decorative element mask map represents the parts that the model expects to retain and change, where the black part represents the part that needs to be changed, and the white part represents the other parts that need to be retained.

[0111] Furthermore, in order to introduce the characteristics of decorative elements, a reference network is added to encode the decorative element images. Optionally, in order to reduce the impact of the newly introduced features on the original Stable Diffusion model, the network structure and parameters of the Reference-Net (reference U-type network) in the reference network can be kept consistent with the U-net (U-type network) in Stable Diffusion. For the diffusion model, its input in the test phase is pure noise, and a meaningful picture is generated after T steps of denoising. The U-net is responsible for predicting the noise between time steps t and t-1. The smaller t is, the less noise is predicted. When t=1, the input of the U-net model is close to noise-free data, and the result after this step of denoising is , after being decoded by the Variational Autodecoder (VAE Decoder), a meaningful image can be generated. Based on the above principles, the decorative element image can be directly input into the Reference-Net after being encoded by the VAE Encoder, and the input t=1 is set. At this time, the reference network encodes the decorative element image, and the encoded features can be used to directly restore the decorative element image. The U-net network mainly includes two structures: ResNet (residual neural network) and Transformer (transformer). ResNet is mainly composed of convolutional networks and is responsible for encoding image features. Transformer is mainly composed of Attention networks, which are responsible for encoding image features and introducing conditional features (such as text, images, and audio features).

[0112] It can be understood that the difference between the incremental training process and the pre-training process is only in the training samples. Figure 3 In the example, the input of the reference network is the first element image of the character decoration element in the complete character portrait at the first angle; the input of the target network is the character image information corresponding to the second angle, such as the second complete character portrait, the partial character portrait with the second element area deleted, and the second mask image. Therefore, the specific process of incremental training refers to the pre-training process and will not be described here.

[0113] In the above embodiment, by introducing the decoration element image as a reference information source to achieve decoration element image replacement, the accuracy of the reference information can be ensured, and then the decoration element replacement result can be ensured to meet expectations, which is conducive to improving work efficiency.

[0114] In an optional embodiment, under the guidance of the decorative element mask map, image features are extracted from the complete character portrait to obtain non-decorative features of areas other than the decorative element area in the complete character portrait, including: image encoding the complete character portrait to obtain a complete latent space representation of the complete character portrait; under the guidance of the decorative element mask map, masking the complete latent space representation to obtain a local latent space representation of areas other than the decorative element area in the complete character portrait; feature encoding and attention pooling processing are performed based on the local latent space representation to obtain non-decorative features of areas other than the decorative element area in the complete character portrait.

[0115] The latent space complete representation refers to the latent space representation of the complete person portrait in the low-dimensional latent space. The latent space local identification refers to the latent space representation of other areas of the complete person portrait except for the decorative elements in the low-dimensional latent space. The latent space representation of an image is an abstract space used to compress and represent image data. The main function of the latent space is to capture the key features of the image by reducing the dimensionality while removing unnecessary details, which enables the model to perform calculations more efficiently and retain important visual information when generating new images.

[0116] Specifically, the server can perform image encoding on the complete person portrait to reduce the dimensionality of the image information and obtain a complete latent space representation of the complete person portrait in a low-dimensional latent space. Then, under the guidance of the decorative element mask map, the complete latent space representation is masked to obtain a latent space local representation of other areas in the complete person portrait except the decorative element area. Finally, the server can obtain non-decorative features of other areas in the complete person portrait except the decorative element area by performing feature encoding and attention pooling on the local latent space representation.

[0117] For example, Fig.10 As shown in the figure, in the Stable Diffusion model, the latent space is used for feature adjustment and noise denoising during the generation process, and finally the data in the latent space is restored to the image in the image space through the decoder. In order to reproduce other areas except the decorative element area, the server can use the complete character portrait as the input of the VAE Encoder to obtain the complete latent space representation of the complete character portrait in the latent space. Then, the decorative element mask map is used to mask the complete representation of the latent space to obtain the local representation of the latent space. After that, the U-Net performs feature encoding and attention pooling on the local representation of the latent space to obtain the non-decorative features of other areas in the complete character portrait except the decorative element area.

[0118] In the above embodiment, the complete character portrait is image encoded to obtain a complete latent space representation of the complete character portrait, which can ensure that the visual information in the complete character portrait is fully mined, and then further masked to obtain a local representation of the latent space, and then obtain non-decorative features, which can ensure the accuracy of the non-decorative features.

[0119] In an optional embodiment, under the guidance of the decorative element mask map, image features are extracted from the complete character portrait to obtain non-decorative features of areas other than the decorative element area in the complete character portrait, including: under the guidance of the decorative element mask map, masking the complete character portrait to obtain a partial character portrait with the decorative element area deleted; image encoding the partial character portrait to obtain a latent space local representation of the partial character portrait; feature encoding and attention pooling processing based on the latent space local representation to obtain non-decorative features of areas other than the decorative element area in the complete character portrait.

[0120] Specifically, the server can first use the decorative element mask image to perform mask processing on the complete person portrait to obtain a partial person portrait with the decorative element area removed. Then, the server can perform image encoding on the partial person portrait to obtain a latent space local representation of the partial person portrait in a low-dimensional latent space. Finally, the server performs feature encoding and attention pooling on the latent space local representation to obtain non-decorative features of other areas in the complete person portrait except the decorative element area. For example, Fig.11 As shown in the figure, the server can use the decorative element mask map to mask the complete person portrait, obtain the partial person portrait after deleting the decorative element area, and then use the partial person portrait as the input of the VAE Encoder to obtain the latent space local representation of the partial person portrait in the low-dimensional latent space. After that, U-Net performs feature encoding and attention pooling on the latent space local representation to obtain the non-decorative features of other areas in the complete person portrait except the decorative element area.

[0121] In the above embodiment, the complete character portrait is first masked to obtain a partial character portrait with the decorative element area deleted, and then image encoding is performed, which can reduce the workload of the image encoding process and is conducive to improving work efficiency.

[0122] In a specific implementation, both the non-decorative features and the decorative element features include query features, key features, and value features. In the case of this embodiment, the training method of the decorative element replacement model also includes: splicing the key features of the non-decorative features and the decorative element features to obtain key fusion features; splicing the value features of the non-decorative features and the decorative element features to obtain value fusion features; based on the query features, key fusion features, and value fusion features in the non-decorative features, obtaining character features.

[0123] Among them, query features (Query, Q), key features (Key, K) and value features (Value, V) play an important role in the attention mechanism, especially in fields such as computer vision and natural language processing. These features are generated by linearly transforming the input and are used to calculate the attention weights to achieve feature fusion and information transmission. The query feature is used to calculate the similarity with the key feature to determine which parts of the features need to be paid attention to. The value feature is used to provide actual feature information so that it can be used to generate output features after attention weighting. In the attention mechanism, the similarity calculation between the query feature and the key feature is usually implemented using the dot product or inner product, and then normalized by the softmax function to obtain the attention weight. These weights are used to weighted sum the value features to generate the final output features. That is:

[0124]

[0125] Among them, d k is a hyperparameter used to represent the dimension of the key feature.

[0126] In Transformer, Attention mainly includes two forms: Self-Attention and Cross-Attention. When Q, K, and V are all image features, it is defined as Self-Attention, which is responsible for calculating the relationship between image features; when Q is an image feature and KV is other conditional features, it is defined as Cross-Attention, which is responsible for calculating the relationship between image features and other features. Studies have shown that the amount of Attention parameters accounts for a small proportion of the entire model, but plays an important role in the model. Taking LORA (Low-Rank Adaptation of Large Language Models) as an example, it only acts on the linear projection part of the Cross-Attention layer Q, K, and V, and the linear part of the subsequent Feedforward Networks (FFN). The amount of fine-tuning parameters is much smaller than all the parameters of U-net, but the fine-tuning effect is similar to fine-tuning the entire U-net parameters.

[0127] Based on this, the present application transfers the Attention feature of Reference-Net to U-net. After U-net receives the feature, it is helpful to generate a highly restored decorative element image. Considering that the Cross-Attention layer involves text features in addition to image features, the introduction of Reference-Net features affects the functioning of text features to a certain extent. In addition, the role of Reference-Net is to encode decorative element images and help U-net restore decorative element images, which only involves image features. Therefore, this embodiment proposes to transfer the Self-Attention feature encoded by Reference-Net to the Self-Attention in U-net in the form of vector splicing. The effect of splicing is as follows: Fig.12 As shown. That is, the server obtains key fusion features by splicing the key features of non-decorative features and decorative element features, and obtains value fusion features by splicing the value features of non-decorative features and decorative element features. Then, based on the query features, key fusion features and value fusion features in the non-decorative features, the character features are obtained. The corresponding formula is:

[0128]

[0129] in, , K and Represents the Self-Attetnion input of U-net, K and It is the Self-Attetnion input of splicing Reference-Net; cat() is the splicing function; y represents the character features.

[0130] In the above embodiment, based on the non-decorative features, only the key features and value features of the decorative element image are spliced. On the one hand, the integrity of the decorative element information can be ensured, and on the other hand, information redundancy can be avoided, which is conducive to improving the replacement effect of the character decorative elements.

[0131] In one embodiment, Fig.13 As shown, a training method for a decoration element replacement model is provided, which can be executed by a computer device, which can be Figure 1 The terminal or server shown, taking the computer device as a server as an example, in this embodiment, the method includes the following steps:

[0132] Step S1301, obtaining multiple candidate character images;

[0133] Step S1302, performing character recognition on each candidate character image to determine the face size of the person included in each candidate character image;

[0134] Step S1303, for each candidate character image, if the face size of the character in the candidate character image meets the size condition and the character includes character decoration elements, a complete character head portrait with the character decoration elements is obtained based on the candidate character image, and a pre-training sample set including multiple complete character head portraits is obtained;

[0135] Among them, the image display effect of the character's decorative elements changes as the angle of the character's head changes;

[0136] Step S1304, for each complete person portrait in the pre-training sample set, identifying the decorative element region in the complete person portrait, and obtaining a decorative element mask map of the complete person portrait;

[0137] Step S1305, extracting image features of the decorative element image in the complete character portrait to obtain decorative element features;

[0138] Step S1306, performing image encoding on the complete person portrait to obtain a complete latent space representation of the complete person portrait;

[0139] Step S1307, under the guidance of the decorative element mask image, mask processing is performed on the complete latent space representation to obtain a latent space partial representation of other areas in the complete character portrait except the decorative element area;

[0140] Step S1308, performing feature encoding and attention pooling processing based on the local representation of the latent space to obtain non-decorative features of other areas in the complete portrait except the decorative element area;

[0141] Among them, both non-decoration features and decoration element features include query features, key features and value features;

[0142] Step S1309, combining the key features of the non-decoration feature and the decoration element feature to obtain a key fusion feature;

[0143] Step S1310, combining the value features of the non-decoration feature and the decoration element feature to obtain a value fusion feature;

[0144] Step S1311, obtaining character features based on the query features, key fusion features and value fusion features in the non-decorative features;

[0145] Step S1312, obtaining an updated image corresponding to the complete character portrait based on character feature decoding;

[0146] Step S1313, when the loss statistics of each complete character portrait converge, a pre-trained model for implementing decoration element image replacement is obtained;

[0147] Among them, the model loss corresponding to any complete person portrait is used to characterize the difference between the complete person portrait and the updated image corresponding to the complete person portrait;

[0148] Step S1314, obtaining a first complete head portrait of the sample person collected at a first angle, and a second complete head portrait of the sample person collected at a second angle;

[0149] Step S1315, identifying a first element region where the character decoration element in the first complete character portrait is located, and obtaining a first mask image for masking the first element region at a first angle;

[0150] Step S1316, normalizing the first mask image to obtain a first normalized mask image;

[0151] The pixel value of the first element region in the first normalized mask image is 1, and the pixel values ​​of other regions except the first element region are 0;

[0152] Step S1317, using the first normalized mask image to perform mask processing on the first complete person portrait, and separating a first element image of a first element region from the first complete person portrait;

[0153] Step S1318, identifying a second element region where the character decoration element in the second complete character portrait is located, and obtaining a second mask image for masking the second element region at a second angle;

[0154] Step S1319, normalizing the second complete person portrait and the second mask image respectively to obtain a normalized portrait and a second normalized mask image;

[0155] The pixel value of the second element region in the second normalized mask image is 0, and the pixel values ​​of other regions except the second element region are 1;

[0156] Step S1320, using the second normalized mask image to perform mask processing on the normalized head portrait to obtain normalized partial head portrait information;

[0157] Step S1321, performing a denormalization process on the local head portrait information to obtain a local head portrait with the second element area deleted;

[0158] Step S1322, constructing an incremental training sample including the second complete person portrait, the first element image, and the partial person portrait with the second element area deleted;

[0159] Step S1323, incrementally training the pre-trained model using the incremental training samples, so that the pre-trained model completes the partial character portrait based on the first element image;

[0160] Step S1324, when the pre-trained model is incrementally trained using a plurality of incremental training samples and a training end condition is satisfied, a decorative element replacement model having an angle compensation capability is obtained;

[0161] Among them, the decoration element replacement model is used to generate the target image of the target person.

[0162] The training method of the decorative element replacement model mentioned above, on the one hand, introduces decorative element images of character decorative elements during the training process, which can solve the problem of weak control ability in using text to describe character decorative elements, avoid the need to repeatedly adjust training samples to achieve training goals, and is conducive to improving work efficiency; on the other hand, through two-stage training, it can ensure that the replacement ability of character decorative elements can be learned in the pre-training stage, and the incremental training stage can solve the generalization problem caused by inconsistent angles, thereby reducing the complexity of the learning process in the pre-training stage with a large number of training samples, and significantly reducing the number of training samples required in the incremental training stage with a higher learning complexity, which is conducive to reducing training costs and also can improve work efficiency.

[0163] In one embodiment, a method for replacing a character decoration element is also provided. The method can be executed by a computer device. The computer device can be Figure 1 The terminal or server shown in FIG. 1 is taken as an example, in this embodiment, the computer device is a terminal. Fig.14 As shown, the method comprises the following steps:

[0164] Step S1402, obtaining a character image of a target character and a reference image including reference decorative elements.

[0165] The target person is the person whose decorative elements need to be replaced, that is, the replacement object of the decorative element replacement. The reference decorative element is the replacement target of the decorative element replacement. That is, to replace the decorative element means to replace the decorative elements originally contained in the target person with the reference decorative elements. The reference image may only contain the reference element image of the reference decorative element, or may contain other images based on the reference element image of the reference decorative element. For example, when the decorative element of the person is a hairstyle, the reference image may be a hairstyle image, or may be a person image with a hairstyle.

[0166] Specifically, the user can initiate a decoration element replacement request through the terminal, so that the server can obtain the character image of the target character and the reference image containing the reference decoration element from the terminal. Furthermore, the character head portrait angles corresponding to the character image and the reference image can be the same or different.

[0167] Step S1404: identifying the decorative element region in the reference image to obtain a reference element image of the reference decorative element.

[0168] The target person has a target decorative element in the person image. The target decorative element is of the same element type as the reference decorative element contained in the reference image. For example, the target decorative element and the reference decorative element are both hairstyles; for another example, the target decorative element and the reference decorative element are both glasses.

[0169] Specifically, the server can identify the decorative element region in the reference image to obtain a reference element image of the reference decorative element. Exemplarily, the server can identify the hairstyle region from a reference character image to obtain a hairstyle image of the reference hairstyle.

[0170] Step S1406: input the reference element image and the character image into the decorative element replacement model to obtain a target image of the target character.

[0171] Specifically, the server uses the reference element image and the character image as inputs of the decorative element replacement model, and can obtain a target image of the target character from the model output. The target character has a reference decorative element in the target image, and the reference decorative element in the target image corresponds to the same character head angle as the target character. The decorative element replacement model is trained based on the decorative element replacement model training method in any of the above decorative element replacement model training method embodiments.

[0172] In an optional embodiment, the decorative element replacement model includes a reference network and a target network. The server can specifically use the reference element image as the input of the reference network and the character image of the target character as the input of the target network, thereby obtaining the target image output by the target network. It can be understood that since the decorative element replacement model obtained through incremental training has the ability of angle compensation, when the angles of the character portraits of the character image and the reference image are different, the angle of the character portrait of the reference decorative element contained in the target image will be consistent with that of the character image, thereby realizing the replacement of decorative elements based on angle compensation.

[0173] In an optional embodiment, take the case where the character decoration element is a hairstyle as an example. Fig.15 As shown, in the case where the decorative element replacement model (hairstyle replacement model) is based on the diffusion model, the model input may include a hairstyle input and a non-hairstyle input, wherein the hairstyle input refers to a reference hairstyle, and the non-hairstyle input may include the original hairstyle mask map of the target person, a local person image with the hairstyle area removed, and a noise map, and the model output is a target image where the original hairstyle is replaced with the reference hairstyle. Fig.15It is not difficult to see that the target image and the reference hairstyle contained in the hairstyle input have different head portrait angles, which leads to differences in image display effects. Compensating for the angle during hairstyle replacement can ensure that the replaced reference hairstyle is consistent with the head portrait angle of the target person in the character image, which is conducive to improving the replacement effect.

[0174] The above-mentioned character decoration element replacement method uses a decoration element replacement model that has been trained in two stages. Without retraining, the original decoration elements originally contained in the target character can be replaced with reference decoration elements while ensuring that other parts of the character remain unchanged, thereby achieving the purpose of zero-shot replacement of character decoration elements, which is conducive to improving work efficiency. In addition, the decoration element replacement model used can solve the generalization problem caused by inconsistent angles, which is conducive to improving the replacement effect.

[0175] In one of the embodiments, the present application also provides an application scenario, in which the above-mentioned decorative element replacement model training method and character decorative element replacement method are applied. The application scenario may be, for example, a hair design scenario. In this application scenario, the character decorative element may be a hairstyle, and during the training of the decorative element replacement model, the server: obtains a pre-trained model for realizing hairstyle image replacement; constructs incremental training samples based on complete character portraits collected for the same person and the same hairstyle at different angles; uses the incremental training samples to incrementally train the pre-trained model, so that the pre-trained model replaces the second hairstyle image in the complete character portrait at a second angle based on the first hairstyle image in the complete character portrait at a first angle; and obtains a hairstyle replacement model with angle compensation capability when the incremental training is completed using multiple incremental training samples. Among them, each pre-trained sample of the pre-trained model includes a complete character portrait with a hairstyle; the image display effect of the hairstyle changes with the change of the angle of the character portrait.

[0176] The trained model can be used for hairstyle replacement. Specifically, the server can obtain a character image of a target character and a reference image containing a reference hairstyle; identify the hairstyle region in the reference image to obtain a hairstyle image of the reference hairstyle; input the hairstyle image and the character image into the hairstyle replacement model to obtain a target image of the target character; the target character has the reference hairstyle in the target image.

[0177] In an optional embodiment, the present application proposes a zero-shot character hairstyle replacement scheme based on a two-stage pre-training based on the Wenshengtu diffusion model. Among them, the two-stage training scheme reduces the data collection and training costs; uses the Inpainting mode of U-net to ensure that only the hairstyle is changed and other parts remain unchanged; introduces Reference-Net to encode hairstyle features, and splices the hairstyle features into the U-net network in the Self-Attention layer to achieve hairstyle replacement. After two-stage pre-training, there is no need to train again. By changing the hairstyle input by Reference-Net, the purpose of zero-shot hairstyle replacement is achieved while ensuring that other parts of the character do not change.

[0178] In one of the embodiments, the present application also provides an application scenario, in which the above-mentioned decorative element replacement model training method and character decorative element replacement method are applied. The application scenario may be, for example, a virtual reality scenario. In this application scenario, the character decorative element may be a helmet, and during the training of the decorative element replacement model, the server: obtains a pre-trained model for realizing helmet image replacement; constructs incremental training samples based on complete character portraits collected from sample characters wearing the same helmet at different angles; uses the incremental training samples to incrementally train the pre-trained model, so that the pre-trained model replaces the second helmet image in the complete character portrait at a second angle based on the first helmet image in the complete character portrait at a first angle; and obtains a helmet replacement model with angle compensation capability when the incremental training is completed using multiple incremental training samples. Among them, each pre-trained sample of the pre-trained model includes a complete character portrait wearing a helmet; the image display effect of the helmet changes with the change of the angle of the character portrait.

[0179] The trained model can be used for helmet replacement. Specifically, the server can obtain a character image of a target character and a reference image including a reference helmet; identify the helmet area in the reference image to obtain a helmet image of the reference helmet; input the helmet image and the character image into the helmet replacement model to obtain a target image of the target character; the target character wears the reference helmet in the target image.

[0180] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0181] Based on the same inventive concept, the embodiment of the present application also provides a training device for a decorative element replacement model for implementing the training method for a decorative element replacement model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more decorative element replacement models provided below can refer to the limitations of the training method for decorative element replacement models above, and will not be repeated here.

[0182] In one embodiment, Fig.16 As shown, a training device for a decoration element replacement model is provided, comprising:

[0183] The pre-trained model acquisition module 1601 is used to acquire a pre-trained model for implementing the replacement of the decorative element image; each pre-trained sample of the pre-trained model includes a complete character portrait with the character decorative element; the image display effect of the character decorative element changes with the change of the character portrait angle;

[0184] An incremental sample construction module 1602 is used to construct incremental training samples based on complete head portraits of the same sample person collected at different angles;

[0185] The incremental training module 1603 is used to perform incremental training on the pre-trained model using the incremental training samples, so that the pre-trained model replaces the second element image of the character decoration element in the complete character portrait at a second angle based on the first element image of the character decoration element in the complete character portrait at a first angle;

[0186] The replacement model determination module 1604 is used to obtain a decoration element replacement model with angle compensation capability by performing incremental training on the pre-trained model using multiple incremental training samples and satisfying the training end condition.

[0187] In one embodiment, the incremental sample construction module 1602 includes: an acquisition unit, used to acquire a first complete character portrait collected for a sample character at a first angle, and a second complete character portrait collected for the sample character at a second angle; a first element region identification unit, used to identify a first element region where a character decorative element is located in the first complete character portrait, and obtain a first mask image for masking the first element region at the first angle; a fusion unit, used to fuse the first complete character portrait and the first mask image, and separate a first element image of the first element region from the first complete character portrait; and a construction unit, used to construct an incremental training sample including the second complete character portrait and the first element image.

[0188] In one of the embodiments, the fusion unit is specifically used to: normalize the first mask image to obtain a first normalized mask image; the pixel value of the first element area in the first normalized mask image is 1, and the pixel values ​​of other areas except the first element area are 0; use the first normalized mask image to mask the first complete character portrait, and separate the first element image of the first element area from the first complete character portrait.

[0189] In one embodiment, the incremental training sample also includes a partial portrait of the person at a second angle; the pre-trained model specifically completes the partial portrait of the person based on the first element image. In the case of this embodiment, the incremental sample construction module 1602 also includes: a second element region identification unit, which is used to identify the second element region where the character decoration element is located in the second complete character portrait, and obtain a second mask map for masking the second element region at a second angle; a normalization unit, which is used to normalize the second complete character portrait and the second mask map respectively, to obtain a normalized portrait and a second normalized mask map; the pixel value of the second element region in the second normalized mask map is 0, and the pixel value of other regions except the second element region is 1; a mask unit, which is used to mask the normalized portrait using the second normalized mask map to obtain normalized partial portrait information; a denormalization unit, which is used to denormalize the partial portrait information to obtain a partial portrait of the person with the second element region deleted.

[0190] In one of the embodiments, the training device of the decorative element replacement model also includes a complete character portrait acquisition module, which is used to: acquire multiple candidate character images; perform character recognition on each candidate character image respectively, and determine the face size of the character contained in each candidate character image; for each candidate character image, if the face size of the character in the candidate character image meets the size condition and the character contains character decorative elements, obtain a complete character portrait with character decorative elements based on the candidate character image.

[0191] In one of the embodiments, the training device of the decorative element replacement model also includes a pre-training model module, which is used to: for each complete character portrait in the pre-training sample set, identify the decorative element area in the complete character portrait to obtain the decorative element mask map of the complete character portrait; perform image feature extraction on the decorative element image in the complete character portrait to obtain the decorative element feature; under the guidance of the decorative element mask map, perform image feature extraction on the complete character portrait to obtain non-decorative features of other areas in the complete character portrait except the decorative element area; based on the character features obtained by fusing the decorative element features and the non-decorative features, decode to obtain an updated image corresponding to the complete character portrait; when the loss statistics of each complete character portrait converge, obtain a pre-training model for realizing decorative element image replacement; the model loss corresponding to any complete character portrait is used to characterize the difference between the complete character portrait and the updated image corresponding to the complete character portrait.

[0192] In one of the embodiments, the pre-trained model module is specifically used to: perform image encoding on the complete character portrait to obtain a complete latent space representation of the complete character portrait; under the guidance of the decorative element mask map, perform mask processing on the complete latent space representation to obtain a latent space local representation of other areas in the complete character portrait except the decorative element area; perform feature encoding and attention pooling processing based on the local latent space representation to obtain non-decorative features of other areas in the complete character portrait except the decorative element area.

[0193] In one of the embodiments, the pre-trained model module is specifically used to: mask the complete character portrait under the guidance of the decorative element mask map to obtain a partial character portrait with the decorative element area deleted; perform image encoding on the partial character portrait to obtain a latent space local representation of the partial character portrait; perform feature encoding and attention pooling processing based on the latent space local representation to obtain non-decorative features of other areas in the complete character portrait except the decorative element area.

[0194] In one embodiment, both the non-decorative features and the decorative element features include query features, key features, and value features. In this embodiment, the pre-trained model module is also used to: concatenate the key features of the non-decorative features and the decorative element features to obtain key fusion features; concatenate the value features of the non-decorative features and the decorative element features to obtain value fusion features; and obtain character features based on the query features, key fusion features, and value fusion features in the non-decorative features.

[0195] Each module in the above-mentioned character decoration element replacement device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0196] Based on the same inventive concept, the embodiment of the present application also provides a character decoration element replacement device for implementing the character decoration element replacement method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more character decoration element replacement device embodiments provided below can refer to the limitations of the character decoration element replacement method above, and will not be repeated here.

[0197] In one embodiment, Fig.17 As shown, a device for replacing a character decoration element is provided, comprising:

[0198] An image acquisition module 1701 is used to acquire a character image of a target character and a reference image including a reference decorative element;

[0199] A decoration element recognition module 1702 is used to recognize a decoration element region in a reference image and obtain a reference element image of a reference decoration element;

[0200] The decorative element replacement module 1703 is used to input the reference element image and the character image into the decorative element replacement model to obtain a target image of the target character; the target character has the reference decorative element in the target image; the decorative element replacement model is trained based on any one of the above-mentioned decorative element replacement model training method embodiments.

[0201] Each module in the above-mentioned character decoration element replacement device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0202] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.18As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the data involved in the above method embodiment. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for a decorative element replacement model or a character decorative element replacement method is implemented.

[0203] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.19 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a training method for a decorative element replacement model or a character decorative element replacement method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0204] Those skilled in the art will understand that Fig.18The structure shown in or 19 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0205] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of any of the above method embodiments when executing the computer program.

[0206] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above method embodiments are implemented.

[0207] In one embodiment, a computer program product is provided, comprising a computer program, which performs the steps of any of the above method embodiments when executed by a processor.

[0208] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions and areas. In addition, the subject can choose not to authorize the subject information and related data, or refuse or conveniently refuse to push information, etc.

[0209] In this application, when collecting and processing relevant data in actual application, the requirements of relevant local laws and regulations should be strictly followed to obtain the informed consent or separate consent of the personal information subject, and subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0210] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0211] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0212] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for a decorative element replacement model, characterized in that: The method comprises: A pre-trained model for implementing the replacement of decorative element images is obtained; each pre-trained sample of the pre-trained model includes a complete head portrait of a person with a decorative element of the person; and the image display effect of the decorative element of the person changes with the change of the angle of the head portrait of the person; Construct incremental training samples based on complete head portraits collected from different angles for the same sample person; Performing incremental training on the pre-trained model using the incremental training samples, so that the pre-trained model replaces the second element image of the character decoration element in the complete character portrait at a second angle based on the first element image of the character decoration element in the complete character portrait at a first angle; When the pre-trained model is incrementally trained using a plurality of incremental training samples and a training end condition is met, a decorative element replacement model with angle compensation capability is obtained; the decorative element replacement model is used to generate a target image of a target person.

2. The method according to claim 1, characterized in that: The step of constructing incremental training samples based on complete head portraits collected from different angles for the same sample person includes: Acquire a first complete head portrait of a sample person collected at a first angle, and a second complete head portrait of the sample person collected at a second angle; Identify a first element region where the character decoration element is located in the first complete character portrait, and obtain a first mask image for masking the first element region at the first angle; fusing the first complete person portrait and the first mask image to separate a first element image of the first element area from the first complete person portrait; Construct incremental training samples including the second complete character portrait and the first element image.

3. The method according to claim 2, characterized in that The fusing the first complete person portrait and the first mask image to separate the first element image of the first element area from the first complete person portrait includes: Normalizing the first mask image to obtain a first normalized mask image; in the first normalized mask image, a pixel value of the first element region is 1, and a pixel value of other regions except the first element region is 0; The first normalized mask image is used to perform mask processing on the first complete person portrait, so as to separate a first element image of the first element area from the first complete person portrait.

4. The method according to claim 2, characterized in that: The incremental training sample also includes a partial head portrait of a person at the second angle; the pre-trained model specifically completes the partial head portrait of a person based on the first element image; The method further comprises: identifying a second element region where the character decoration element is located in the second complete character portrait, and obtaining a second mask image for masking the second element region at the second angle; Normalizing the second complete person portrait and the second mask image respectively to obtain a normalized portrait and a second normalized mask image; in the second normalized mask image, the pixel value of the second element region is 0, and the pixel value of other regions except the second element region is 1; Using the second normalized mask image to perform mask processing on the normalized head portrait to obtain normalized local head portrait information; The partial head portrait information is denormalized to obtain a partial head portrait with the second element area deleted.

5. The method according to claim 1, characterized in that: The process of obtaining the complete character portrait includes: Acquire multiple candidate person images; Performing character recognition on each of the candidate character images to determine the face size of the person included in each of the candidate character images; For each of the candidate character images, if the face size of the character in the candidate character image meets the size condition and the character includes character decoration elements, a complete character portrait with the character decoration elements is obtained based on the candidate character image.

6. The method according to any one of claims 1 to 5, characterized in that The training process of the pre-trained model includes: For each complete person portrait in the pre-training sample set, identifying a decorative element region in the complete person portrait, and obtaining a decorative element mask map of the complete person portrait; Extracting image features of the decorative element image in the complete portrait to obtain decorative element features; Under the guidance of the decorative element mask image, image feature extraction is performed on the complete person portrait to obtain non-decorative features of other areas of the complete person portrait except the decorative element area; Based on the character features obtained by fusing the decorative element features and the non-decorative features, decoding to obtain an updated image corresponding to the complete character portrait; When the loss statistics of each of the complete character portraits converge, a pre-trained model for realizing decorative element image replacement is obtained; the model loss corresponding to any of the complete character portraits is used to characterize the difference between the complete character portrait and the updated image corresponding to the complete character portrait.

7. The method according to claim 6, characterized in that Under the guidance of the decorative element mask image, image feature extraction is performed on the complete person portrait to obtain non-decorative features of other areas of the complete person portrait except the decorative element area, including: Performing image encoding on the complete person portrait to obtain a complete latent space representation of the complete person portrait; Under the guidance of the decorative element mask map, mask processing is performed on the complete latent space representation to obtain a latent space local representation of other regions in the complete person portrait except the decorative element region; Feature encoding and attention pooling processing are performed based on the local representation of the latent space to obtain non-decorative features of other areas in the complete character portrait except the decorative element area.

8. The method according to claim 6, characterized in that Under the guidance of the decorative element mask image, image feature extraction is performed on the complete person portrait to obtain non-decorative features of other areas of the complete person portrait except the decorative element area, including: Under the guidance of the decorative element mask image, mask processing is performed on the complete character portrait to obtain a partial character portrait with the decorative element area deleted; Performing image encoding on the partial head portrait of the person to obtain a latent space partial representation of the partial head portrait of the person; Feature encoding and attention pooling processing are performed based on the local representation of the latent space to obtain non-decorative features of other areas in the complete character portrait except the decorative element area.

9. The method according to claim 6, characterized in that The non-decoration feature and the decoration element feature both include a query feature, a key feature and a value feature; the method further includes: splicing the key features of the non-decorative feature and the decorative element feature to obtain a key fusion feature; splicing the value features of the non-decoration feature and the decoration element feature to obtain a value fusion feature; Based on the query feature in the non-decoration feature, the key fusion feature and the value fusion feature, a character feature is obtained.

10. A method for replacing character decoration elements, characterized in that: The method comprises: Acquire a character image of a target character and a reference image including reference decorative elements; identifying a decorative element region in the reference image to obtain a reference element image of the reference decorative element; The reference element image and the character image are input into a decorative element replacement model to obtain a target image of the target character; the target character has the reference decorative element in the target image; the decorative element replacement model is trained based on the method described in any one of claims 1 to 9.

11. A training device for a decorative element replacement model, characterized in that: The device comprises: A pre-trained model acquisition module is used to acquire a pre-trained model for implementing decorative element image replacement; each pre-trained sample of the pre-trained model includes a complete character portrait with a character decorative element; the image display effect of the character decorative element changes with the change of the character portrait angle; An incremental sample construction module, used to construct incremental training samples based on complete head portraits collected from different angles for the same sample person; An incremental training module, configured to perform incremental training on the pre-trained model using the incremental training samples, so that the pre-trained model replaces a second element image of the character decoration element in the complete character portrait at a second angle based on a first element image of the character decoration element in the complete character portrait at a first angle; The decorative element replacement model determination module is used to obtain a decorative element replacement model with angle compensation capability by incrementally training the pre-trained model using multiple incremental training samples and satisfying the training end conditions; the decorative element replacement model is used to generate a target image of a target person.

12. A device for replacing character decoration elements, characterized in that: The device comprises: An image acquisition module, used to acquire a character image of a target character and a reference image including reference decorative elements; A decorative element recognition module, used to recognize a decorative element region in the reference image and obtain a reference element image of the reference decorative element; A decorative element replacement module is used to input the reference element image and the character image into a decorative element replacement model to obtain a target image of the target character; the target character has the reference decorative element in the target image; the decorative element replacement model is trained based on the method described in any one of claims 1 to 9.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images