Image processing method, face replacement model processing method, device and electronic equipment

Abstract

The present application relates to artificial intelligence technology, and is characterized by an image processing method, a face replacement model processing method, a device and an electronic device, the method comprising: extracting object attribute features from face images of different target real objects, the object attribute features comprising posture features, expression features and lighting features of the target real objects; based on the virtual character attribute features learned during the training process, sequentially decoding the posture features and expression features in each of the object attribute features to obtain different target synthetic images; the object in each of the target synthetic images has a posture and expression consistent with the corresponding target real object, and has identity information consistent with the target virtual character; according to each of the lighting features, the corresponding target synthetic image is illuminated and rendered. The use of this method can simplify the process of face replacement and improve the authenticity of the synthetic image after the face is better.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method, a face replacement model processing method, a device and an electronic device. Background Art

[0002] With the continuous development of image processing technology, users can replace the faces in their own or other people's images, such as transferring some facial features of a virtual character to the user's own or other people's images, thereby achieving the purpose of face replacement.

[0003] In traditional face replacement schemes, a 3D human model is usually first photographed to obtain a model image, and images of other people are photographed to capture different expressions and postures, and then the model person in the model image is replaced with the face in the person image to obtain a composite image. However, the above scheme requires a special 3D human model, which makes face replacement complicated, and the composite image obtained by face replacement based on the 3D human model has poor authenticity. Summary of the invention

[0004] Based on this, it is necessary to provide an image processing method, a face replacement model processing method, a device and an electronic device to address the above technical problems, which can simplify the face replacement process and improve the authenticity of the synthesized image after the face is better.

[0005] An image processing method, the method comprising:

[0006] Extracting object attribute features from face images of different target real objects, wherein the object attribute features include posture features, expression features, and illumination features of the target real objects;

[0007] Based on the virtual character attribute features learned during the training process, the posture features and expression features in the attribute features of each object are decoded in turn to obtain different target synthetic images; the object in each target synthetic image has a posture and expression consistent with the corresponding target real object, and has identity information consistent with the target virtual character;

[0008] According to each of the illumination features, illumination rendering is performed on the corresponding target synthetic image.

[0009] An image processing device, comprising:

[0010] An extraction module, used to extract object attribute features from face images of different target real objects, wherein the object attribute features include posture features, expression features and illumination features of the target real objects;

[0011] A decoding module, for decoding the posture features and expression features in the attribute features of each object in turn based on the attribute features of the virtual character learned during the training process, to obtain different target synthetic images; the object in each target synthetic image has a posture and expression consistent with the corresponding target real object, and has identity information consistent with the target virtual character;

[0012] A rendering module is used to perform lighting rendering on the corresponding target composite image according to each of the lighting features.

[0013] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0014] Extracting object attribute features from face images of different target real objects, wherein the object attribute features include posture features, expression features, and illumination features of the target real objects;

[0015] Based on the virtual character attribute features learned during the training process, the posture features and expression features in the attribute features of each object are decoded in turn to obtain different target synthetic images; the object in each target synthetic image has a posture and expression consistent with the corresponding target real object, and has identity information consistent with the target virtual character;

[0016] According to each of the illumination features, illumination rendering is performed on the corresponding target synthetic image.

[0017] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0018] Extracting object attribute features from face images of different target real objects, wherein the object attribute features include posture features, expression features, and illumination features of the target real objects;

[0019] Based on the virtual character attribute features learned during the training process, the posture features and expression features in the attribute features of each object are decoded in turn to obtain different target synthetic images; the object in each target synthetic image has a posture and expression consistent with the corresponding target real object, and has identity information consistent with the target virtual character;

[0020] According to each of the illumination features, illumination rendering is performed on the corresponding target synthetic image.

[0021] The above-mentioned image processing method, device, electronic device and storage medium extract object attribute features from each face image to extract posture features, expression features and lighting features of each target real object; based on the virtual character attribute features learned during the training process, the posture features and expression features in each object attribute feature are decoded in turn, so that different target synthetic images can be obtained, wherein the object in each target synthetic image has a posture and expression consistent with the corresponding target real object, and has identity information consistent with the target virtual character, so that there is no need to specially make a three-dimensional human body model, simplifying the process of face replacement; and because the target synthetic image is obtained based on the virtual character attribute features, posture features and expression features learned during the training process, rather than replacing the face based on the manufactured three-dimensional human body model, the authenticity of the target synthetic image can be effectively improved. In addition, based on the virtual character attribute features learned during the training process, the posture features and expression features of multiple target real objects can be decoded to obtain different target synthetic images, thereby realizing the image face replacement of multiple users using the attribute features of a virtual character, which is conducive to the maintenance of the virtual character image. Finally, the corresponding target composite image is rendered according to the various lighting features, which can further improve the authenticity of the target composite image, avoid the need for a lot of post-production of the rendering results in traditional solutions, and simplify the process of face replacement.

[0022] A method for processing a face replacement model, the method comprising:

[0023] Extracting identity features of real objects from face image samples;

[0024] Extracting attribute features from the face image sample and the virtual character image in sequence through a universal encoder to obtain training object attribute features and training virtual character attribute features respectively;

[0025] Decoding the identity feature and the training object attribute feature by a universal decoder to obtain a first composite image; and decoding the training virtual character attribute feature by a dedicated decoder to obtain a second composite image;

[0026] According to the difference value between the first synthetic image and the human face image sample, adjusting the parameters of the general encoder and the general decoder; and according to the loss value between the second synthetic image and the virtual character image, adjusting the parameters of the general encoder and the dedicated decoder;

[0027] When the model converges, a face replacement model is formed based on the universal encoder and the dedicated decoder; the face replacement model is used to perform face replacement on face images of different target real objects.

[0028] A face replacement model processing device, the device comprising:

[0029] A first extraction module is used to extract identity features of a real object from a face image sample;

[0030] A second extraction module is used to extract attribute features from the face image samples and the virtual character images in sequence through a universal encoder to obtain training object attribute features and training virtual character attribute features respectively;

[0031] A decoding module, configured to decode the identity feature and the training object attribute feature by a universal decoder to obtain a first composite image; and to decode the training virtual character attribute feature by a dedicated decoder to obtain a second composite image;

[0032] an adjustment module, configured to adjust parameters of the universal encoder and the universal decoder according to a difference value between the first synthetic image and the human face image sample; and to adjust parameters of the universal encoder and the dedicated decoder according to a loss value between the second synthetic image and the virtual character image;

[0033] A building module is used to form a face replacement model based on the universal encoder and the dedicated decoder when the model converges; the face replacement model is used to perform face replacement on face images of different target real objects.

[0034] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Extracting identity features of real objects from face image samples;

[0036] Extracting attribute features from the face image sample and the virtual character image in sequence through a universal encoder to obtain training object attribute features and training virtual character attribute features respectively;

[0037] Decoding the identity feature and the training object attribute feature by a universal decoder to obtain a first composite image; and decoding the training virtual character attribute feature by a dedicated decoder to obtain a second composite image;

[0038] According to the difference value between the first synthetic image and the human face image sample, adjusting the parameters of the general encoder and the general decoder; and according to the loss value between the second synthetic image and the virtual character image, adjusting the parameters of the general encoder and the dedicated decoder;

[0039] When the model converges, a face replacement model is formed based on the universal encoder and the dedicated decoder; the face replacement model is used to perform face replacement on face images of different target real objects.

[0040] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0041] Extracting identity features of real objects from face image samples;

[0042] Extracting attribute features from the face image sample and the virtual character image in sequence through a universal encoder to obtain training object attribute features and training virtual character attribute features respectively;

[0043] Decoding the identity feature and the training object attribute feature by a universal decoder to obtain a first composite image; and decoding the training virtual character attribute feature by a dedicated decoder to obtain a second composite image;

[0044] According to the difference value between the first synthetic image and the human face image sample, adjusting the parameters of the general encoder and the general decoder; and according to the loss value between the second synthetic image and the virtual character image, adjusting the parameters of the general encoder and the dedicated decoder;

[0045] When the model converges, a face replacement model is formed based on the universal encoder and the dedicated decoder; the face replacement model is used to perform face replacement on face images of different target real objects.

[0046] The above-mentioned face replacement model processing method, device, electronic device and storage medium extract attribute features from face image samples through a universal encoder, decode the identity features and training object attribute features extracted from the face image samples through a universal decoder, and adjust the parameters of the universal encoder and the universal decoder according to the difference value between the decoded first synthetic image and the face image sample, so that the universal encoder can better extract the object attribute features from the face image of the target real object, so that the object attribute features are retained during the face replacement process, while the identity features of the target real object are hidden. In addition, the attribute features of the virtual character image are extracted through the universal encoder, and the extracted training virtual character attribute features are decoded through the dedicated decoder. According to the loss value between the second synthetic image and the virtual character image, the parameters of the universal encoder and the dedicated decoder are adjusted, so that the virtual character attribute features can be learned, so as to achieve face replacement of the face image of the target real object based on the learned virtual character attribute features and object attribute features, which can effectively improve the authenticity of the target synthetic image and simplify the face replacement process. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A diagram showing an application environment of an image processing method and a face replacement model processing method in one embodiment;

[0048] Figure 2 is a schematic flow chart of an image processing method in one embodiment;

[0049] Figure 3 is a schematic diagram of facial feature points in one embodiment;

[0050] Figure 4 A schematic diagram of performing face replacement processing on a face image of a target real object by using a face replacement model in one embodiment;

[0051] Figure 5 It is a flowchart of a method for processing a face replacement model in one embodiment;

[0052] Figure 6 A schematic diagram of the structure of model training in one embodiment;

[0053] Figure 7 A schematic diagram of a process of obtaining a hybrid generator and generating a virtual character image based on the hybrid generator in one embodiment;

[0054] Figure 8 A schematic diagram of a hybrid generator obtained by replacing network parameters of corresponding network layers in a model generator and a star generator in one embodiment;

[0055] Fig. 9 A schematic diagram for comparing corresponding images generated by different generators in one embodiment;

[0056] Fig.10 is a structural block diagram of an image processing device in one embodiment;

[0057] Fig.11 is a structural block diagram of a face replacement model processing device in one embodiment;

[0058] Fig.12 is a structural block diagram of a face replacement model processing device in another embodiment;

[0059] Fig.13 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0062] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0063] Among them, computer vision technology (CV) is a science that studies how to make machines "see", usually including image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional (3D) object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and also includes common biometric recognition technologies such as face recognition and fingerprint recognition.

[0064] Machine Learning (ML) is a study of how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning usually includes artificial neural networks, belief networks, reinforcement learning, deep learning, transfer learning, inductive learning, and self-learning.

[0065] The solution provided in the embodiments of the present application involves artificial intelligence computer vision and machine learning technologies, which are specifically described by the following embodiments:

[0066] The image processing method and face replacement model processing method provided by this application can be applied to Figure 1In the application environment shown. In the application environment, a terminal 102 and a server 104 are included. The face replacement model processing method can be applied to the server 104, and the image processing method can be applied to the terminal 102 or the server 104. Specifically, the server 104 extracts the identity features of the real object from the face image sample; extracts the attribute features of the face image sample and the virtual character image in turn through the universal encoder to obtain the training object attribute features and the training virtual character attribute features respectively; decodes the identity features and the training object attribute features through the universal decoder to obtain a first synthetic image; and, decodes the training virtual character attribute features through the dedicated decoder to obtain a second synthetic image; adjusts the parameters of the universal encoder and the universal decoder according to the difference value between the first synthetic image and the face image sample; and adjusts the parameters of the universal encoder and the dedicated decoder according to the loss value between the second synthetic image and the virtual character image; when the model converges, a face replacement model is formed based on the universal encoder and the dedicated decoder, and then the face replacement model is deployed on the terminal 102.

[0067] When the image processing method can be applied to terminal 102, terminal 102 extracts object attribute features from the facial image of the target real object in response to a replacement operation initiated by the user, the object attribute features including posture features, expression features and lighting features of the target real object; based on the virtual character attribute features learned during the training process, the posture features and expression features in the object attribute features are decoded in turn to obtain a target synthetic image; the object in the target synthetic image has a posture and expression consistent with the target real object, and has identity information consistent with the target virtual character; based on the lighting features, the target synthetic image is illuminated and rendered, so as to obtain and display the final synthetic image of the replaced face.

[0068] In addition, when the image processing method can be applied to the server 104, the terminal 102 sends the facial image of the target real object to the server 104, and the server 104 extracts object attribute features from the facial image of the target real object, and the object attribute features include posture features, expression features and lighting features of the target real object; based on the virtual character attribute features learned during the training process, the posture features and expression features in the object attribute features are decoded in turn to obtain a target composite image; the object in the target composite image has a posture and expression consistent with the target real object, and has identity information consistent with the target virtual character; based on the lighting features, the target composite image is illuminated and rendered to obtain a final composite image with the face replaced, and the final composite image is sent to the terminal 102 for display.

[0069] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, etc., but is not limited thereto.

[0070] The server 104 may be an independent physical server or a service node in a blockchain system. The service nodes in the blockchain system form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP).

[0071] In addition, server 104 can also be a server cluster composed of multiple physical servers, and can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0072] The terminal 102 and the server 104 may be connected via Bluetooth, USB (Universal Serial Bus) or a network or other communication connection methods, which are not limited in this application.

[0073] In one embodiment, Figure 2 As shown, an image processing method is provided, which can be applied to Figure 1 The terminal 102 in the embodiment can also be applied to Figure 1 In the following embodiment, the method is applied to the server 104. Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0074] S202, extracting object attribute features from face images of different target real objects.

[0075] The target real object may refer to a real user whose face image needs to be replaced. The face image may refer to an image containing facial features, such as an image obtained by only capturing the face of the target real object, or an image captured simultaneously with the face and other body parts of the target real object, such as an image containing both the face of the target real object and the upper and lower body of the target real object.

[0076] In addition, object attribute features are used to reflect the non-identity features of the target real object, including the posture features, expression features and lighting features of the target real object. The posture feature may refer to the features that characterize the head posture of the target real object, such as raising the head, lowering the head and tilting the head. Expression features may refer to facial features used to express emotions and feelings. Different expression features can usually be presented through a combination of different facial organs, such as a happy expression feature can be presented by slightly squinting the eyes and slightly opening the mouth. Lighting features may refer to features used to reflect the intensity and direction of lighting on the face of the target real object.

[0077] In one embodiment, the server may receive facial images of different target real objects taken by each terminal, and then extract object attribute features in the facial images. Alternatively, the server extracts target video frames containing facial features of target real objects from user videos to be published or film and television videos to be played, and then extracts object attribute features in each target video frame through a universal encoder; wherein the target video frames extracted from different user videos or different film and television videos belong to facial images of different target real objects.

[0078] In one embodiment, the server performs posture recognition on the face images of different target real objects through a universal encoder to identify whether the target real object is in a state of raising its head, lowering its head, or tilting its head, and obtains the posture feature of the target real object, thereby completing the extraction of the posture feature. In addition, the server can also perform face recognition on the face images of different target real objects through a universal encoder, and can identify the facial feature points of the different target real objects, obtain the corresponding expression features, and thus complete the extraction of the expression features.

[0079] like Figure 3 As shown, face recognition on face images can be obtained as follows Figure 3 The recognition results of facial feature points shown in the figure are marked with numbers for the convenience of explanation. For example, Figure 3 1 to 17 shown in the figure represent facial edge feature points, 18 to 22 and 23 to 27 represent the user's left eyebrow feature points and right eyebrow feature points respectively, 28 to 36 represent the user's nose feature points, 37 to 42 represent the user's left eye feature points, 43 to 48 represent the user's right eye feature points, and 49 to 68 represent the user's lip feature points. By combining the identified feature points, the expression features can be obtained. For example, the combination of slightly squinting eye feature points and slightly open lip feature points can obtain the smiling expression features.

[0080] In addition, the server can also intercept the target area containing the face in the face image through the universal encoder, determine the maximum brightness feature, minimum brightness feature and illumination direction feature of the face of the target real object in the target area, and thus obtain the illumination feature. The illumination feature includes the maximum brightness feature, the minimum brightness feature and the illumination direction feature.

[0081] S204, based on the virtual character attribute features learned during the training process, the posture features and expression features in the attribute features of each object are decoded in turn to obtain different target synthetic images.

[0082] Among them, the object in the target synthetic image can also be called a synthetic object or a face-changed object. The object in each target synthetic image has a posture and expression consistent with the corresponding target real object, and has identity information consistent with the target virtual character. For example, assuming that the target real object is user A, and the target virtual character is a virtual character that does not exist in the real world, then the object in the target synthetic image is a character that combines the posture features and expression features of user A and the identity information of the virtual character. Visually, the object in the target synthetic image has a posture and expression consistent with user A, but because the object has identity information consistent with the virtual character, it cannot be recognized that the object is user A during face recognition.

[0083] In one embodiment, S204 may specifically include: after the server extracts the object attribute features through the general encoder, the server inputs the object attribute features into a dedicated decoder, so that the dedicated decoder decodes the posture features and expression features in each object attribute feature in turn based on the virtual character attribute features learned during the training process to obtain different target composite images.

[0084] Among them, the virtual character attribute features may include the identity information, posture features, expression features and lighting features of the virtual character (i.e., virtual object). A general encoder refers to an encoder that can extract attribute features from the face image of any target real object. A dedicated decoder refers to an encoder that can decode based on the extracted object attribute features of the target real object to obtain a target synthetic image, and the object in the target synthetic image has a posture and expression consistent with the target real object, as well as identity information consistent with the virtual character.

[0085] Specifically, after the server inputs the object attribute features into the dedicated decoder, the dedicated decoder decodes the posture features and expression features in each object attribute feature in turn based on the identity information in the virtual character attribute features to obtain different target composite images.

[0086] In one embodiment, a universal encoder and a dedicated decoder are two parts of a face replacement model, and the universal encoder and the dedicated decoder are obtained by joint training with a universal decoder. The specific training method includes: extracting identity features of real objects from face image samples; extracting attribute features of face image samples and virtual character images in turn through a universal encoder to obtain training object attribute features and training virtual character attribute features respectively; decoding the identity features and training object attribute features through a universal decoder to obtain a first synthetic image; and, decoding the training virtual character attribute features through a dedicated decoder to obtain a second synthetic image; adjusting the parameters of the universal encoder and the universal decoder according to the difference value between the first synthetic image and the face image sample; and adjusting the parameters of the universal encoder and the dedicated decoder according to the loss value between the second synthetic image and the virtual character image; when the model converges, a face replacement model is formed based on the universal encoder and the dedicated decoder. Among them, the virtual character image can also be called a virtual portrait, which is a two-dimensional image of a virtual character. The virtual character can also be called a virtual object, which can specifically refer to a virtual person.

[0087] In one embodiment, a server obtains a training set of facial images with different characteristics; the training sets of facial images with different characteristics are respectively input into different generators for training; after the training is completed, based on the network parameters of at least one layer in the first target generator, the network parameters of the corresponding layer in the second target generator are replaced to obtain a hybrid generator; both the first target generator and the second target generator are trained generators; and a virtual character image is generated through the hybrid generator.

[0088] In one embodiment, the first target generator is used to generate an image with a first characteristic; the second target generator is used to generate an image with a second characteristic. The above step of generating a virtual character image by a hybrid generator may specifically include: a server obtains an image to be converted; and inputs the image to be converted into the hybrid generator, so that a network layer in the hybrid generator performs image processing on the image to be converted, thereby obtaining a virtual character image with the first characteristic and the second characteristic.

[0089] In one embodiment, the step of obtaining the image to be converted may specifically include: the server obtains at least two images to be converted captured under different lighting conditions; the real objects in each image to be converted have different postures and expressions. Correspondingly, the step of performing image processing on the image to be converted to obtain a virtual character image having the first characteristic and the second characteristic may specifically include: the server sequentially performs image processing on the at least two images to be converted to obtain at least two virtual character images having the first characteristic and the second characteristic; wherein the virtual character in each virtual character image has a different posture and expression.

[0090] In one embodiment, the first feature and the second feature are different ages, or different skin colors, or different dressing styles; or, the first feature and the second feature are a combination of age, skin color, and dressing style.

[0091] In one embodiment, the step of replacing the network parameters of the corresponding layer in the second target generator based on the network parameters of at least one layer in the first target generator may specifically include: the server determines the network layer of the parameters to be replaced in the second target generator; determines the network layer that matches the network layer of the parameters to be replaced in the first target generator; and replaces the network parameters of the network layer of the parameters to be replaced based on the network parameters of the matching network layer in the first target generator.

[0092] In one embodiment, the universal encoder includes a first universal encoder and a second universal encoder that share network parameters. Correspondingly, the above-mentioned step of extracting attribute features from the face image sample and the virtual character image in sequence through the universal encoder may specifically include: the server normalizes the face image sample and the virtual character image respectively; inputs the processed face image sample into the first universal encoder so that the first universal encoder extracts attribute features from the face image sample; and extracts attribute features from the processed virtual character image through the second universal encoder.

[0093] In one embodiment, the difference value between the first synthetic image and the face image sample includes a first difference value and a second difference value. Correspondingly, the above step of adjusting the parameters of the universal encoder and the universal decoder according to the difference value between the first synthetic image and the face image sample may specifically include: the server determines the first difference value according to the first synthetic image and the face image sample; extracts image features from the first synthetic image and the face image sample respectively to obtain the first synthetic image feature and the face image feature; determines the second difference value according to the first synthetic image feature and the face image feature; and adjusts the parameters of the universal encoder and the universal decoder according to the first difference value and the second difference value.

[0094] In one embodiment, the loss value between the second synthetic image and the virtual character image includes a first loss value, a second loss value, and a third loss value. Correspondingly, the above-mentioned step of adjusting the parameters of the general encoder and the dedicated decoder according to the loss value between the second synthetic image and the virtual character image may specifically include: determining the first loss value according to the second synthetic image and the virtual character image; extracting image features from the second synthetic image and the virtual character image respectively to obtain second synthetic image features and virtual character image features; determining the second loss value according to the second synthetic image features and the virtual character image features; discriminating the second synthetic image and the virtual character image respectively through a discriminator to obtain a first discrimination result and a second discrimination result; determining the third loss value according to the first discrimination result and the second discrimination result; and adjusting the parameters of the general encoder and the dedicated decoder according to the first loss value, the second loss value, and the third loss value.

[0095] S206: Perform illumination rendering on the corresponding target synthetic image according to each illumination feature.

[0096] Specifically, the server arranges lighting probes at the face of the object in the corresponding target synthetic image according to the lighting characteristics of each target real object, and then performs lighting rendering on the corresponding target synthetic image according to the lighting probes, so that the lighting conditions can be well rendered even if the real object is in motion.

[0097] In order to better understand the solution of this application, a specific scenario is used for illustration, as follows: when a user publishes a personal video or personal image in a public place, the user can select his or her favorite virtual character (ie, virtual person) to replace it. Figure 4 As shown, the general encoder extracts the user's attribute features (the attribute features include posture features, expression features and lighting features) from the user's personal video or personal image, and then the dedicated decoder decodes the user's posture features and expression features based on the virtual character attribute features for the virtual character learned during the training process, and then performs lighting rendering on the decoded target synthetic image according to the lighting features to obtain the final target synthetic image, which retains the user's own expression features, posture features and lighting features, hides the user's own identity information, and replaces it with the identity information of the virtual character, thereby maintaining the authenticity of the personal video and personal image without leaking one's own identity information. For example, before publishing his dance video, user A can transfer the identity information of his favorite virtual idol to the user's face in the dance video, and then publish it, so that viewers can see that the "character" in the dance video has the dance posture, head posture and expression of user A, and the face of the "character" looks like the virtual idol of user A.

[0098] In the above embodiment, object attribute features are extracted from each face image to extract posture features, expression features and lighting features of each target real object; based on the virtual character attribute features learned during the training process, the posture features and expression features in each object attribute feature are decoded in turn, and different target synthetic images can be obtained, wherein the object in each target synthetic image has a posture and expression consistent with the corresponding target real object, and has identity information consistent with the target virtual character, so there is no need to specially make a three-dimensional human body model, simplifying the process of face replacement; and because the target synthetic image is obtained based on the virtual character attribute features, posture features and expression features learned during the training process, rather than replacing the face based on the manufactured three-dimensional human body model, the authenticity of the target synthetic image can be effectively improved. In addition, based on the virtual character attribute features learned during the training process, the posture features and expression features of multiple target real objects can be decoded to obtain different target synthetic images, thereby realizing the image face replacement of multiple users using the attribute features of a virtual character, which is conducive to the maintenance of the virtual character image. Finally, the corresponding target composite image is rendered according to the various lighting features, which can further improve the authenticity of the target composite image, avoid the need for a lot of post-production of the rendering results in traditional solutions, and simplify the process of face replacement.

[0099] In one embodiment, Figure 5 As shown, a face replacement model processing method is provided, and the method is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0100] S502, extracting identity features of the real object from the face image sample.

[0101] The real object may refer to a real user. The face image sample may refer to an image captured with facial features for model training, such as an image captured only by the face of the target real object, or an image captured simultaneously by the face and other body parts of the target real object.

[0102] Identity features, which may also be referred to as identity information, are features used to represent the identity of a real object, such as iris features, nose features, and lip features of the real object.

[0103] In one embodiment, the server extracts identity features from the face image sample through a face recognition model to obtain the identity features of the real object, such as Figure 6 As shown in Figure (a).

[0104] S504, extracting attribute features from the face image samples and the virtual character images in sequence through a universal encoder to obtain training object attribute features and training virtual character attribute features respectively.

[0105] The universal encoder refers to an encoder that can extract attribute features of both face image samples and virtual character images, and after training, the universal encoder can also extract attribute features of face images of target real objects.

[0106] The training object attribute features may refer to the attribute features about the real object extracted from the face image samples during the training phase. The training object attribute features are used to reflect the non-identity features of the real object, including the posture features, expression features and lighting features of the real object. The posture features in the training object attribute features may refer to the features that characterize the head posture of the real object, such as raising the head, lowering the head and tilting the head. The expression features in the training object attribute features may refer to the facial features used to express emotions and feelings. Different expression features can usually be presented through a combination of different facial organs, such as a happy expression feature can be presented by slightly squinting the eyes and slightly opening the mouth. The lighting features in the training object attribute features may refer to the features used to reflect the intensity and direction of lighting on the face of the real object.

[0107] The training virtual character attribute features may refer to the attribute features for the virtual character extracted from the virtual character image during the training phase, and the training virtual character attribute features may include the identity features, expression features, posture features, and lighting features of the virtual character. The identity features, expression features, posture features, and lighting features of the virtual character are similar to the relevant features of the real object, and the specific introduction may refer to the relevant features of the real object.

[0108] In one embodiment, the universal encoder includes a first universal encoder and a second universal encoder that share network parameters. Therefore, the step of extracting attribute features from the face image sample and the virtual character image in sequence by the universal encoder in S504 may specifically include: the server normalizes the face image sample and the virtual character image respectively; inputs the processed face image sample into the first universal encoder, so that the first universal encoder extracts attribute features from the face image sample, such as Figure 6 As shown in Figure (a); the attribute features of the processed virtual character image are extracted by the second universal encoder, such as Figure 6 As shown in Figure (b).

[0109] The specific steps for normalizing the facial image samples may include: the server may perform global pooling and at least two convolutions on the facial image samples through the estimation network to obtain the predicted mean and predicted variance of the facial image samples, and then perform normalization on the facial image samples based on the predicted mean and predicted variance. Alternatively, the server may directly calculate the statistical mean and statistical variance of the facial image samples, and then perform normalization on the facial image samples based on the statistical mean and statistical variance. Alternatively, the predicted mean, statistical mean, predicted variance and statistical variance of the facial image samples are obtained in the above manner, and then the predicted mean and statistical mean are weighted to obtain a weighted mean; and the predicted variance and statistical variance are weighted to obtain a weighted variance; and the facial image samples are normalized according to the weighted mean and weighted variance.

[0110] In addition, for the specific steps of normalizing the virtual character image, reference may be made to the steps of normalizing the above-mentioned human face image samples.

[0111] In one embodiment, before or before the normalization process, the server may also perform denoising on the face image samples and the virtual character images to prevent noise points from affecting the accuracy of the model.

[0112] S506, decoding the identity features and the training object attribute features by a universal decoder to obtain a first synthetic image; and decoding the training virtual character attribute features by a dedicated decoder to obtain a second synthetic image.

[0113] Among them, the universal decoder refers to a decoder that can decode based on the extracted identity features and training object attribute features to obtain a decoder similar to the face image sample, so that the universal encoder can better extract posture features, expression features and lighting features.

[0114] The dedicated decoder is a decoder that can decode based on the extracted training virtual character attribute features of the virtual character to obtain a second synthetic image, and the object in the second synthetic image has a posture, expression and identity information consistent with the virtual character. In addition, after the training is completed, the dedicated decoder can decode based on the object attribute features of the target real object extracted by the universal encoder to obtain a target synthetic image, and the object in the target synthetic image has a posture and expression consistent with the target real object, and identity information consistent with the virtual character.

[0115] In one embodiment, the step of decoding the identity features and the training object attribute features by a universal decoder in S506 may specifically include: the server inputs the identity features and the training object attribute features into the universal decoder, so that the universal decoder decodes the posture features and the expression features in the training object attribute features based on the identity features to obtain a first initial synthetic image; then, the server performs lighting rendering on the first initial synthetic image based on the lighting features in the training object attribute features to obtain a first synthetic image.

[0116] In one embodiment, the training virtual character attribute features include the virtual character's identity features, posture features, expression features and lighting features; therefore, the step of decoding the training virtual character attribute features through a dedicated decoder in S506 may specifically include: the server inputs the training virtual character attribute features into the dedicated decoder, so that the dedicated decoder decodes the posture features and expression features based on the virtual character's identity features to obtain a second initial synthetic image; then, the server performs lighting rendering on the second initial synthetic image based on the lighting features in the training virtual character attribute features to obtain a second synthetic image.

[0117] S508, adjusting parameters of the general encoder and the general decoder according to the difference value between the first synthetic image and the face image sample; and adjusting parameters of the general encoder and the dedicated decoder according to the loss value between the second synthetic image and the virtual character image.

[0118] The difference value may refer to the similarity between the first composite image and the face image sample, and the smaller the difference value, the greater the similarity. Similarly, the loss value may refer to the similarity between the second composite image and the virtual character image, and the smaller the loss value, the greater the similarity.

[0119] In order to more clearly describe the parameter adjustment of the above codec, the following description is divided into two parts:

[0120] (1) Parameter adjustment of the universal encoder and universal decoder.

[0121] In one embodiment, the server determines the difference value between the first synthetic image and the face image sample, and then back-propagates the difference value in the universal encoder and the universal decoder to obtain the gradient value of the network parameters of each network layer in the universal encoder and the universal decoder, and adjusts the network parameters of the network layer based on the gradient value. In addition, when the universal encoder includes the first universal encoder, the server may also back-propagate the difference value in the first universal encoder and the universal decoder to obtain the gradient value of the network parameters of each network layer in the first universal encoder and the universal decoder, and adjust the network parameters of the network layer based on the gradient value.

[0122] Specifically, the difference value between the first synthetic image and the face image sample includes a first difference value and a second difference value. The step of adjusting parameters of the universal encoder and the universal decoder according to the difference value between the first synthetic image and the face image sample in S508 may specifically include: the server determines the first difference value according to the first synthetic image and the face image sample; extracts image features from the first synthetic image and the face image sample respectively to obtain first synthetic image features and face image features; determines the second difference value according to the first synthetic image features and the face image features; and adjusts parameters of the universal encoder and the universal decoder according to the first difference value and the second difference value.

[0123] For example, the server may calculate the first difference value between the first synthetic image and the face image sample by using an L1 loss function, where the L1 loss function is:

[0124] L 1other =abs(Dec other (Enc(Ohter i ), Identity(Ohter i ))-(Ohter i ))

[0125] Among them, Ohter i It can refer to the i-th face image sample, Identity (Ohter i ) can refer to the identity features extracted by the face recognition model, Enc(Ohter i ) can refer to the attribute features of the training object extracted by the universal encoder, and Dec other (Enc(Ohter i ), Identity(Ohter i )) may refer to the first composite image decoded by the universal decoder, and abs() refers to the absolute value of the difference between the first composite image and the i-th face image sample.

[0126] In addition, the server can also use L LPIPS The loss function calculates the second difference value between the first synthetic image and the face image sample, the L LPIPS The loss function is:

[0127] L LPIPSothe =[LPIPS(Dec other (Enc(Ohter i ), Identity(Ohter i )))-LPIPS(Ohter i )] 2

[0128] Among them, LPIPS (Ohter i ) represents the features extracted from the i-th face image sample, LPIPS (Dec other (Enc(Ohter i ), Identity(Ohter i ))) represents the features extracted from the first synthetic image, so L LPIPSothe It can represent the similarity between the first synthetic image and the face image sample in the feature space (ie, the second difference value).

[0129] When the universal encoder includes a first universal encoder, the server determines a difference and a value between a first difference value and a second difference value, and then inputs the difference and the value into the first universal encoder and the universal decoder for back propagation to obtain gradient values ​​of network parameters of each network layer in the first universal encoder and the universal decoder, and adjusts the network parameters of the network layer based on the gradient values.

[0130] (2) Parameter adjustment of general encoder and dedicated decoder.

[0131] In one embodiment, the server determines the loss value between the second synthetic image and the virtual character image, and then back-propagates the loss value in the general encoder and the dedicated decoder to obtain the gradient value of the network parameters of each network layer in the general encoder and the dedicated decoder, and adjusts the network parameters of the network layer based on the gradient value. In addition, when the general encoder also includes a second general encoder, the server can also back-propagate the loss value in the second general encoder and the dedicated decoder to obtain the gradient value of the network parameters of each network layer in the second general encoder and the dedicated decoder, and adjust the network parameters of the network layer based on the gradient value.

[0132] Specifically, the loss value between the second synthetic image and the virtual character image includes a first loss value, a second loss value and a third loss value; therefore, the step of adjusting parameters of the general encoder and the dedicated decoder according to the loss value between the second synthetic image and the virtual character image in S508 may specifically include: the server determines a first loss value according to the second synthetic image and the virtual character image; extracts image features from the second synthetic image and the virtual character image respectively to obtain second synthetic image features and virtual character image features; determines a second loss value according to the second synthetic image features and the virtual character image features; discriminates the second synthetic image and the virtual character image respectively through a discriminator to obtain a first discrimination result and a second discrimination result; determines a third loss value according to the first discrimination result and the second discrimination result; and adjusts parameters of the general encoder and the dedicated decoder according to the first loss value, the second loss value and the third loss value.

[0133] For example, the server may calculate the first loss value between the second synthesized image and the virtual character image by using an L1 loss function, where the L1 loss function is:

[0134] L 1A =abs(Dec A (Enc(A i )))-(A i ))

[0135] Among them, A i represents the i-th virtual character image, Enc(A i ) represents the training virtual character attribute features extracted by the universal encoder from the i-th virtual character image, Dec A (Enc(A i )) represents the second synthetic image obtained by decoding the attribute features of the training virtual character by the dedicated decoder, and abs() refers to the absolute value of the difference between the second synthetic image and the i-th virtual character image.

[0136] In addition, the server can also use L LPIPS The loss function calculates the second loss value, the L LPIPS The loss function is:

[0137] L LPIPSA =[LPIPS(Dec A (Enc(A i )))-LPIPS(A i )] 2

[0138] Among them, LPIPS (A i ) features extracted from the i-th virtual character image, LPIPS (Dec A (Enc(A i ))) represents the features extracted from the second synthetic image, and L LPIPSA It can represent the similarity between the second synthetic image and the i virtual character images in the feature space (ie, the second loss value).

[0139] Finally, the server can also use L GAN The loss function calculates the third loss value, the L GAN The loss function is as follows:

[0140]

[0141] Among them, DA j (Dec A (Enc(A i )))) represents the first discrimination result obtained by the j-th discriminator on the second synthetic image, DAj (A i ) represents the second discrimination result obtained by the j-th discriminator on the i-th virtual character image, and E() represents the calculated mean.

[0142] When the universal encoder also includes a second universal encoder, the server determines the loss sum value between the first loss value, the second loss value and the third loss value, and then inputs the loss sum value into the second universal encoder and the dedicated decoder for back propagation to obtain the gradient values ​​of the network parameters of each network layer in the second universal encoder and the dedicated decoder, and adjusts the network parameters of the network layer based on the gradient values.

[0143] S510, when the model converges, a face replacement model is formed based on the universal encoder and the dedicated decoder; the face replacement model is used to perform face replacement on face images of different target real objects.

[0144] In one embodiment, when the model converges, a face replacement model is constructed based on a general encoder and a dedicated decoder, and then the face replacement model is deployed on a server or terminal so that when the user needs to replace the face in the image, the face replacement model can be used to replace the face of the facial image.

[0145] In the above embodiment, attribute features are extracted from face image samples by a universal encoder, identity features and training object attribute features extracted from face image samples are decoded by a universal decoder, and parameters of the universal encoder and universal decoder are adjusted according to the difference value between the decoded first synthetic image and the face image sample, so that the universal encoder can better extract object attribute features from the face image of the target real object, so that the object attribute features are retained during the face-changing process, while the identity features of the target real object are hidden. In addition, attribute features are extracted from virtual character images by a universal encoder, and the extracted training virtual character attribute features are decoded by a dedicated decoder, and parameters of the universal encoder and dedicated decoder are adjusted according to the loss value between the second synthetic image and the virtual character image, so that the virtual character attribute features can be learned, so that the face image of the target real object can be replaced based on the learned virtual character attribute features and object attribute features, which can effectively improve the authenticity of the target synthetic image and simplify the face replacement process.

[0146] In one embodiment, Figure 7 As shown, the method also includes:

[0147] S702, obtaining a training set of face images with different characteristics.

[0148] The different characteristics may be different ages, different skin colors or different dressing styles. For example, the face image training set with different characteristics may be a face image training set of real objects of different ages, or a face image training set of real objects of different skin colors, or a face image training set of different dressing styles. The face image training set with different dressing styles may include a face image training set of celebrity style and a face image training set of model style.

[0149] S704, inputting the face image training sets with different characteristics into different generators for training.

[0150] In one embodiment, the server inputs the training sets of facial images of real objects of different ages into different generators for training, thereby obtaining dedicated generators for generating facial images of different ages. In addition, the server inputs the training sets of facial images of real objects of different skin colors into different generators for training, thereby obtaining dedicated generators for generating facial images of different skin colors. Finally, the server inputs the training sets of facial images of real objects of different dressing styles into different generators for training, thereby obtaining dedicated generators for generating facial images of different dressing styles.

[0151] For example, the server inputs celebrity-style face images into the generator for training, thereby obtaining a dedicated generator for generating celebrity-style face images (referred to as celebrity generator). The server inputs model-style face images into the generator for training, thereby obtaining a dedicated generator for generating model-style face images (referred to as model generator).

[0152] S706, after completing the training, based on the network parameters of at least one layer in the first target generator, the network parameters of the corresponding layer in the second target generator are replaced to obtain a hybrid generator; the first target generator and the second target generator are both trained generators.

[0153] In one embodiment, S706 may specifically include: the server determines the network layer of the parameters to be replaced in the second target generator; determines the network layer that matches the network layer of the parameters to be replaced in the first target generator; and replaces the network parameters of the network layer of the parameters to be replaced based on the network parameters of the matching network layer in the first target generator.

[0154] Among them, the replacement of network parameters in the network layer can be understood as replacing the corresponding part of the network layer in the second target generator with the network layer of the first target generator.

[0155] For example, assuming that the first target generator is a star generator for generating star-style images, and the second target generator is a model generator for generating model-style images, the network parameters of the i-th network layer and subsequent network layers of the model generator can be replaced with the network parameters of the i-th network layer and subsequent network layers of the star generator, thereby obtaining a hybrid generator that mixes the star generator and the model generator, such as Figure 8 shown.

[0156] S708, generating a virtual character image through a hybrid generator.

[0157] In one embodiment, the first target generator is used to generate an image with a first characteristic; the second target generator is used to generate an image with a second characteristic. Correspondingly, S708 may specifically include: the server obtains the image to be converted; inputs the image to be converted into the hybrid generator, so that the network layer in the hybrid generator performs image processing on the image to be converted, and obtains a virtual character image with the first characteristic and the second characteristic.

[0158] The first characteristic and the second characteristic are different ages, or different skin colors, or different dressing styles; or, the first characteristic and the second characteristic are a combination of age, skin color and dressing style.

[0159] In one embodiment, the above-mentioned step of obtaining the image to be converted may specifically include: the server obtains at least two images to be converted captured under different lighting conditions; the real objects in each image to be converted have different postures and expressions; performing image processing on the image to be converted to obtain a virtual character image having a first characteristic and a second characteristic includes: performing image processing on at least two images to be converted in sequence to obtain at least two virtual character images having the first characteristic and the second characteristic; wherein the virtual character in each virtual character image has a different posture and expression.

[0160] like Fig. 9 As shown, the star generator can generate star images, the model generator can generate model images, and the hybrid generator can generate virtual character images, which mix the features of model images and star images. Therefore, the hybrid generator can mix the features of the faces of objects in different images to obtain an image of a virtual character (i.e., a virtual character image).

[0161] In the above embodiment, by replacing the parameters in the corresponding network layer of the second target generator, the obtained hybrid generator can be used to generate virtual character images of interest to users, thereby enriching the characteristics and style of the virtual character images and meeting the needs of different users for different virtual character images.

[0162] The present application also provides different application scenarios, such as privacy protection application scenarios, game and film production application scenarios, and virtual idol maintenance application scenarios, which can be applied to the above-mentioned image processing method. The application of the image processing method in these application scenarios is as follows:

[0163] Scenario 1: Application scenario of privacy protection

[0164] When users post personal videos in public, they can choose their favorite virtual characters to replace them. That is, the user's facial expression characteristics, posture characteristics and lighting characteristics are retained, the user's identity information is hidden, and replaced with the identity information of the virtual character. This can maintain the authenticity of the personal video without leaking their own identity information.

[0165] Scenario 2: Application scenario of game and film production

[0166] In the process of film and television production, after designing the virtual character image, you only need to build the scene and ask the actors to perform to obtain the film and television image containing the face, and then convert the attribute characteristics of the actor in the film and television image into the virtual character image to complete the face-changing process. The actor's expression characteristics, posture characteristics and lighting characteristics are retained, the actor's identity information is hidden, and replaced with the identity information of the virtual character. There is no need for a lot of post-production, which can greatly simplify the pipeline of computer animation (Computer Graphics, CG) production in real-life scenes.

[0167] Scenario 3: Application scenario of virtual idol maintenance

[0168] Nv1 virtual human video face swapping can be realized, that is, a specific virtual human can be replaced with any actor, realizing the unbinding of virtual idols and actors so as to maintain the virtual idols.

[0169] Combined with the above application scenarios, the specific face-changing process is further described as follows:

[0170] 1. Virtual Portrait Generation

[0171] Use multiple pre-trained stylegan generators to generate virtual portraits without the need to design a 3D human model:

[0172] (1) Collect multiple portrait datasets with characteristics, such as children's image collection, model image collection, celebrity image collection, etc. In addition, you can also obtain portrait datasets with different characteristics by subclassifying the open source FFHQ dataset;

[0173] (2) Use portrait datasets with different characteristics to train the StyleGAN generator separately, so as to fine-tune the network of the corresponding network layer in the StyleGAN generator and obtain a dedicated generator for images of specific styles or characteristics;

[0174] (3) Using the layer exchange technology of StyleGAN, the network parameters of different scales of different dedicated generators are combined to obtain a hybrid generator. This hybrid generator can be used to generate virtual portraits with different characteristics, as shown below: Figure 2 , the two model parameters are mixed as follows:

[0175] P inter =(1-a)P star +aP model

[0176]

[0177] Among them, P interp It can represent the network parameters obtained by mixing the network parameters of a certain network layer, r represents the number of layers of the network layer in the dedicated generator, and r swap Indicates the condition that the number of layers satisfies. For example, assuming that the model generator and the star generator have a total of 18 network layers, then when r swap =9, the network parameters of the 10th to 18th network layers in the model generator are replaced by the network parameters of the 10th to 18th network layers in the star generator, thereby obtaining a hybrid generator.

[0178] (4) According to different scene requirements, more dedicated generators can be trained to mix network parameters, ensuring that different users can obtain virtual portraits of interest.

[0179] (II) Training of face replacement model

[0180] Among them, the face replacement model can support the Nv1 replacement mode, that is, using a virtual portrait to replace the face images of multiple users.

[0181] For the face-changing solution of Nv1, a decoder (i.e., a dedicated decoder) is specially designed for a specific virtual character to greatly improve the effect of face-changing, as follows:

[0182] (1) Using the attribute editing function of stylegan, the generated virtual portrait A0 is expanded to generate virtual portraits with multiple poses, multiple expressions, and multiple lighting conditions, and multiple virtual portraits Ai are obtained by manual screening based on identity consistency;

[0183] (2) In addition to the above virtual portraits Ai, we also collect arbitrary portraits Otheri;

[0184] (3) When training the model, you can use Figure 6 The model structure is trained, and the model structure includes a general encoder, a general decoder, and a dedicated decoder. Among them, the general encoder is responsible for encoding the posture, expression, and lighting information of any portrait, the general decoder is responsible for generating portraits of any identity, and the dedicated decoder is only responsible for generating virtual portrait A. In addition, the model structure also includes a face recognition model Identity and multi-scale discriminators DA1, DA2, and DA3 for the virtual portrait A. Among them, the face recognition model is used to extract the identity features of the real user image.

[0185] (4) Calculate the loss function

[0186] like Figure 6 As shown in Figure (a), the losses in this part include L1 and LPIPS. Through a large number of image training, the universal encoder can better extract information such as posture, expression and lighting:

[0187] L 1othe =abs(Dec other (Enc(Ohter i ), Identity(Ohter i ))-(Ohter i ))

[0188] L LPIPSothe =[LPIPS(Dec other (Enc(Ohter i ), Identity(Ohter i )))-LPIPS(Ohter i )] 2

[0189] like Figure 6 As shown in Figure (b), for the reconstruction of the virtual portrait A, in addition to calculating the reconstruction loss, there is also an adversarial loss, the purpose of which is to enhance the quality of the generated virtual character A. This dedicated decoder will be used for subsequent deployment.

[0190] L 1A =abs(Dec A (Enc(A i )))-(A i ))

[0191] L LPIPSA =[LPIPS(DeC A (Enc(A i )))-LPIPS(A i )] 2

[0192]

[0193] The total training loss is:

[0194] L total =L 1other +L LPIPSothe +L 1A +L LPIPSA +L GAN

[0195] 3. Deployment of face replacement model

[0196] When deploying the model, you only need to use the above-mentioned general encoder and special decoder to form a face replacement model for deployment. When you need to replace the face of any user's face image or video frame in a video, input the face image or the corresponding video frame into the face replacement model, and through the processing of the face replacement model, you can get a virtual portrait (the virtual person in the virtual portrait has the same posture, expression and lighting as the user, and has the same identity information as the virtual person), thereby completing the face replacement.

[0197] The above embodiments can achieve the following technical effects:

[0198] (1) By combining the generation of 2D virtual portraits with the Nv1 face-swapping method, high-quality face-swapping of images and videos is achieved;

[0199] (2) The generation of 2D virtual portraits is cheaper, faster, and more diverse than CG production;

[0200] (3) When using Nv1's face-changing solution, you only need to shoot videos or images naturally in real scenes. There are no special requirements for the actors. Compared with CG production, there is no need to use motion capture and expression capture equipment, and no need for complex post-production. It is low-cost and high-efficiency. In addition, the obtained synthetic images are more realistic, avoiding the "uncanny valley" effect commonly seen in CG production.

[0201] It should be understood that although Figure 2 , 5 The steps in the flowchart of 7 are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , 5At least part of the steps in 7 may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0202] In one embodiment, Fig.10 As shown, an image processing device is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of an electronic device. The device specifically includes: an extraction module 1002, a decoding module 1004 and a rendering module 1006, wherein:

[0203] Extraction module 1002, used to extract object attribute features from face images of different target real objects, the object attribute features including posture features, expression features and illumination features of the target real objects;

[0204] The decoding module 1004 is used to decode the posture features and expression features in the attribute features of each object in turn based on the attribute features of the virtual character learned during the training process, so as to obtain different target synthetic images; the object in each target synthetic image has a posture and expression consistent with the corresponding target real object, and has identity information consistent with the target virtual character;

[0205] The rendering module 1006 is used to perform lighting rendering on the corresponding target synthetic image according to various lighting features.

[0206] In one embodiment, the extraction module 1002 is further used to extract a target video frame containing facial features of a target real object from a user video to be published or a film and television video to be played; wherein the target video frames extracted from different user videos or different film and television videos belong to face images of different target real objects;

[0207] The extraction module 1002 is further configured to extract object attribute features in each target video frame through a universal encoder.

[0208] In one embodiment, the decoding module 1004 is also used to input the object attribute features into the dedicated decoder after the general encoder extracts the object attribute features, so that the dedicated decoder decodes the posture features and expression features in each object attribute feature in turn based on the virtual character attribute features learned during the training process.

[0209] In the above embodiment, object attribute features are extracted from each face image to extract posture features, expression features and lighting features of each target real object; based on the virtual character attribute features learned during the training process, the posture features and expression features in each object attribute feature are decoded in turn, and different target synthetic images can be obtained, wherein the object in each target synthetic image has a posture and expression consistent with the corresponding target real object, and has identity information consistent with the target virtual character, so there is no need to specially make a three-dimensional human body model, simplifying the process of face replacement; and because the target synthetic image is obtained based on the virtual character attribute features, posture features and expression features learned during the training process, rather than replacing the face based on the manufactured three-dimensional human body model, the authenticity of the target synthetic image can be effectively improved. In addition, based on the virtual character attribute features learned during the training process, the posture features and expression features of multiple target real objects can be decoded to obtain different target synthetic images, thereby realizing the image face replacement of multiple users using the attribute features of a virtual character, which is conducive to the maintenance of the virtual character image. Finally, the corresponding target composite image is rendered according to the various lighting features, which can further improve the authenticity of the target composite image, avoid the need for a lot of post-production of the rendering results in traditional solutions, and simplify the process of face replacement.

[0210] For the specific definition of the image processing device, please refer to the definition of the image processing method above, which will not be repeated here. Each module in the above-mentioned image processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0211] In one embodiment, Fig.11 As shown, a face replacement model processing device is provided, which can be a part of an electronic device using a software module or a hardware module, or a combination of the two. The device specifically includes: a first extraction module 1102, a second extraction module 1104, a decoding module 1106, an adjustment module 1108 and a construction module 1110, wherein:

[0212] A first extraction module 1102 is used to extract identity features of a real object from a face image sample;

[0213] The second extraction module 1104 is used to extract attribute features from the face image samples and the virtual character images in sequence through a universal encoder to obtain training object attribute features and training virtual character attribute features respectively;

[0214] The decoding module 1106 is used to decode the identity feature and the training object attribute feature by a universal decoder to obtain a first composite image; and decode the training virtual character attribute feature by a dedicated decoder to obtain a second composite image;

[0215] An adjustment module 1108 is used to adjust parameters of a general encoder and a general decoder according to a difference value between the first synthetic image and the face image sample; and to adjust parameters of a general encoder and a dedicated decoder according to a loss value between the second synthetic image and the virtual character image;

[0216] The construction module 1110 is used to form a face replacement model based on the universal encoder and the dedicated decoder when the model converges; the face replacement model is used to perform face replacement on face images of different target real objects.

[0217] In the above embodiment, attribute features are extracted from face image samples by a universal encoder, identity features and training object attribute features extracted from face image samples are decoded by a universal decoder, and parameters of the universal encoder and universal decoder are adjusted according to the difference value between the decoded first synthetic image and the face image sample, so that the universal encoder can better extract object attribute features from the face image of the target real object, so that the object attribute features are retained during the face-changing process, while the identity features of the target real object are hidden. In addition, attribute features are extracted from virtual character images by a universal encoder, and the extracted training virtual character attribute features are decoded by a dedicated decoder, and parameters of the universal encoder and dedicated decoder are adjusted according to the loss value between the second synthetic image and the virtual character image, so that the virtual character attribute features can be learned, so that the face image of the target real object can be replaced based on the learned virtual character attribute features and object attribute features, which can effectively improve the authenticity of the target synthetic image and simplify the face replacement process.

[0218] In one embodiment, Fig.12 As shown, the device also includes:

[0219] An acquisition module 1112 is used to acquire a training set of face images with different characteristics;

[0220] An input module 1114 is used to input facial image training sets with different characteristics into different generators for training;

[0221] A replacement module 1116 is used to replace the network parameters of the corresponding layer in the second target generator based on the network parameters of at least one layer in the first target generator after the training is completed, so as to obtain a hybrid generator; the first target generator and the second target generator are both trained generators;

[0222] The generation module 1118 is used to generate a virtual character image through a hybrid generator.

[0223] In one embodiment, the first target generator is used to generate an image having a first characteristic; the second target generator is used to generate an image having a second characteristic;

[0224] The generation module 1118 is also used to obtain the image to be converted; input the image to be converted into the hybrid generator so that the network layer in the hybrid generator performs image processing on the image to be converted to obtain a virtual character image with the first characteristic and the second characteristic.

[0225] In one embodiment, the generation module 1118 is further used to obtain at least two images to be converted that are captured under different lighting conditions; the real objects in each image to be converted have different postures and expressions; the at least two images to be converted are processed in turn to obtain at least two virtual character images having a first characteristic and a second characteristic; wherein the virtual character in each virtual character image has a different posture and expression.

[0226] In one embodiment, the first feature and the second feature are different ages, or different skin colors, or different dressing styles; or, the first feature and the second feature are a combination of age, skin color, and dressing style.

[0227] In one embodiment, the replacement module 1116 is also used to determine the network layer of the parameters to be replaced in the second target generator; determine the network layer that matches the network layer of the parameters to be replaced in the first target generator; and replace the network parameters of the network layer of the parameters to be replaced based on the network parameters of the matching network layer in the first target generator.

[0228] In one embodiment, the universal encoder includes a first universal encoder and a second universal encoder that share network parameters;

[0229] The second extraction module 1104 is also used to normalize the face image samples and the virtual character images respectively; input the processed face image samples into the first universal encoder so that the first universal encoder extracts attribute features from the face image samples; and extract attribute features from the processed virtual character images through the second universal encoder.

[0230] In one embodiment, the difference value between the first synthetic image and the face image sample includes a first difference value and a second difference value;

[0231] The adjustment module 1108 is also used to determine a first difference value based on the first synthetic image and the face image sample; extract image features from the first synthetic image and the face image sample respectively to obtain first synthetic image features and face image features; determine a second difference value based on the first synthetic image features and the face image features; and adjust parameters of the universal encoder and the universal decoder based on the first difference value and the second difference value.

[0232] In one embodiment, the loss value between the second synthetic image and the virtual character image includes a first loss value, a second loss value, and a third loss value;

[0233] The adjustment module 1108 is also used to determine a first loss value based on the second synthetic image and the virtual character image; extract image features from the second synthetic image and the virtual character image respectively to obtain second synthetic image features and virtual character image features; determine a second loss value based on the second synthetic image features and the virtual character image features; discriminate the second synthetic image and the virtual character image respectively through a discriminator to obtain a first discrimination result and a second discrimination result; determine a third loss value based on the first discrimination result and the second discrimination result; and adjust parameters of the general encoder and the dedicated decoder based on the first loss value, the second loss value and the third loss value.

[0234] In the above embodiment, by replacing the parameters in the corresponding network layer of the second target generator, the obtained hybrid generator can be used to generate virtual character images of interest to users, thereby enriching the characteristics and style of the virtual character images and meeting the needs of different users for different virtual character images.

[0235] The specific definition of the face replacement model processing device can be found in the definition of the face replacement model processing method above, which will not be repeated here. Each module in the above-mentioned face replacement model processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0236] In one embodiment, an electronic device is provided. The electronic device may be a terminal or a server. Taking the device as a server as an example, its internal structure diagram may be as follows: Fig.13As shown. The electronic device includes a processor, a memory and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store facial images, facial image samples and virtual character images. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an image processing method or a face replacement model processing method is implemented.

[0237] Those skilled in the art will understand that Fig.13 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0238] In one embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0239] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0240] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the steps in the above-mentioned method embodiments.

[0241] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0242] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0243] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: The method comprises: Extracting object attribute features from face images of different target real objects through a universal encoder of the face replacement model, wherein the object attribute features include posture features, expression features and illumination features of the target real objects; By using a dedicated decoder of the face replacement model, based on the virtual character attribute features learned during the training process, the posture features and expression features in the attribute features of each object are decoded in turn to obtain different target synthetic images; the object in each target synthetic image has a posture and expression consistent with the corresponding target real object, and has identity information consistent with the target virtual character; According to each of the illumination features, performing illumination rendering on the corresponding target synthetic image; Among them, the method of training the face replacement model is: extracting attribute features of face image samples and virtual character images in turn through a universal encoder to obtain training object attribute features and training virtual character attribute features respectively; decoding the identity features of the face image samples and the training object attribute features through a universal decoder to obtain a first synthetic image; and decoding the training virtual character attribute features through a dedicated decoder to obtain a second synthetic image; adjusting the parameters of the universal encoder and the universal decoder according to the difference value between the first synthetic image and the face image sample; and adjusting the parameters of the universal encoder and the dedicated decoder according to the loss value between the second synthetic image and the virtual character image until the model converges; the training object attribute features and the training virtual character attribute features are the attribute features for the real object and the attribute features for the virtual character extracted in the training stage respectively; The method for generating the virtual character image is as follows: obtaining a training set of facial images with different characteristics; inputting the training sets of facial images with different characteristics into different generators for training; after completing the training, based on the network parameters of at least one layer in the first target generator, replacing the network parameters of the corresponding layer in the second target generator to obtain a hybrid generator; the first target generator and the second target generator both belong to the trained generator; and generating the virtual character image through the hybrid generator.

2. The method according to claim 1, characterized in that The method further comprises: Extracting a target video frame containing facial features of the target real object from a user video to be published or a film and television video to be played; Among them, the target video frames extracted from different user videos or different film and television videos belong to face images of different target real objects.

3. The method according to claim 2, characterized in that The dedicated decoder of the face replacement model sequentially decodes the posture features and expression features in each of the object attribute features based on the virtual character attribute features learned during the training process, including: After the universal encoder extracts the object attribute features, the object attribute features are input into the dedicated decoder, so that the dedicated decoder decodes the posture features and expression features in each of the object attribute features in turn based on the virtual character attribute features learned during the training process.

4. A face replacement model processing method, characterized in that: The method comprises: Extracting identity features of real objects from face image samples; Extracting attribute features from the face image samples and the virtual character images in sequence through a universal encoder to obtain training object attribute features and training virtual character attribute features, respectively; the training object attribute features and the training virtual character attribute features are attribute features for the real object and attribute features for the virtual character extracted in the training phase, respectively; Decoding the identity feature and the training object attribute feature by a universal decoder to obtain a first composite image; and decoding the training virtual character attribute feature by a dedicated decoder to obtain a second composite image; According to the difference value between the first synthetic image and the human face image sample, adjusting the parameters of the general encoder and the general decoder; and according to the loss value between the second synthetic image and the virtual character image, adjusting the parameters of the general encoder and the dedicated decoder; When the model converges, a face replacement model is formed based on the universal encoder and the dedicated decoder; the face replacement model is used to perform face replacement on face images of different target real objects; Among them, the method for generating the virtual character image is: obtaining a training set of facial images with different characteristics; inputting the training sets of facial images with different characteristics into different generators for training; after completing the training, based on the network parameters of at least one layer in the first target generator, replacing the network parameters of the corresponding layer in the second target generator to obtain a hybrid generator; the first target generator and the second target generator both belong to the trained generator; and generating the virtual character image through the hybrid generator.

5. The method according to claim 4, characterized in that Extracting the identity features of the real object from the face image sample comprises: The identity features of the face image samples are extracted through the face recognition model to obtain the identity features of the real object; the identity features include iris features, nose features and lip features.

6. The method according to claim 4, characterized in that The first target generator is used to generate an image with a first characteristic; the second target generator is used to generate an image with a second characteristic; Generating the virtual character image by the hybrid generator comprises: Get the image to be converted; The image to be converted is input into the hybrid generator, so that the network layer in the hybrid generator performs image processing on the image to be converted to obtain a virtual character image having the first characteristic and the second characteristic.

7. The method according to claim 6, characterized in that The obtaining of the image to be converted comprises: Acquire at least two images to be converted that are captured under different illuminations; the real object in each of the images to be converted has different postures and expressions; The performing image processing on the image to be converted to obtain a virtual character image having the first characteristic and the second characteristic comprises: performing image processing on at least two of the to-be-converted images in sequence to obtain at least two virtual character images having the first characteristic and the second characteristic; The virtual character in each of the virtual character images has different postures and expressions.

8. The method according to claim 6, characterized in that The first characteristic and the second characteristic are different ages, or different skin colors, or different dressing styles; or, The first feature and the second feature are combinations of age, skin color and dressing style.

9. The method according to claim 4, characterized in that The replacing of the network parameters of the corresponding layer in the second target generator based on the network parameters of at least one layer in the first target generator includes: Determine, in the second target generator, a network layer of parameters to be replaced; Determining, in the first target generator, a network layer that matches the network layer of the parameter to be replaced; Based on the network parameters of the network layer matched in the first target generator, the network parameters of the network layer whose parameters are to be replaced are replaced.

10. The method according to claim 4, characterized in that The universal encoder includes a first universal encoder and a second universal encoder that share network parameters; The extracting attribute features of the face image sample and the virtual character image in sequence by using a universal encoder comprises: Performing normalization processing on the face image sample and the virtual character image respectively; Inputting the processed face image sample into the first universal encoder so that the first universal encoder extracts attribute features from the face image sample; The attribute features of the processed virtual character image are extracted by the second universal encoder.

11. The method according to any one of claims 4 to 10, characterized in that: The difference value between the first synthetic image and the face image sample includes a first difference value and a second difference value; The adjusting parameters of the universal encoder and the universal decoder according to the difference value between the first synthetic image and the face image sample comprises: Determine a first difference value according to the first composite image and the face image sample; Extracting image features from the first synthetic image and the facial image sample respectively to obtain first synthetic image features and facial image features; determining a second difference value according to the first synthetic image features and the facial image features; Parameters of the universal encoder and the universal decoder are adjusted according to the first difference value and the second difference value.

12. The method according to any one of claims 4 to 10, characterized in that: The loss value between the second synthetic image and the virtual character image includes a first loss value, a second loss value and a third loss value; The adjusting parameters of the general encoder and the dedicated decoder according to the loss value between the second synthetic image and the virtual character image comprises: determining a first loss value according to the second composite image and the virtual character image; Extracting image features from the second synthetic image and the virtual character image respectively to obtain second synthetic image features and virtual character image features; determining a second loss value according to the second synthetic image features and the virtual character image features; Using a discriminator to discriminate the second synthetic image and the virtual character image respectively, to obtain a first discrimination result and a second discrimination result; determining a third loss value according to the first discrimination result and the second discrimination result; Parameters of the universal encoder and the dedicated decoder are adjusted according to the first loss value, the second loss value and the third loss value.

13. An image processing device, characterized in that: The device comprises: An extraction module, used for extracting object attribute features from face images of different target real objects through a universal encoder of a face replacement model, wherein the object attribute features include posture features, expression features and illumination features of the target real objects; A decoding module, used to decode the posture features and expression features in the attribute features of each object in turn through a dedicated decoder of the face replacement model based on the attribute features of the virtual character learned during the training process, so as to obtain different target synthetic images; the object in each target synthetic image has a posture and expression consistent with the corresponding target real object, and has identity information consistent with the target virtual character; A rendering module, used for performing illumination rendering on the corresponding target composite image according to each of the illumination features; Among them, the method of training the face replacement model is: extracting attribute features of face image samples and virtual character images in turn through a universal encoder to obtain training object attribute features and training virtual character attribute features respectively; decoding the identity features of the face image samples and the training object attribute features through a universal decoder to obtain a first synthetic image; and decoding the training virtual character attribute features through a dedicated decoder to obtain a second synthetic image; adjusting the parameters of the universal encoder and the universal decoder according to the difference value between the first synthetic image and the face image sample; and adjusting the parameters of the universal encoder and the dedicated decoder according to the loss value between the second synthetic image and the virtual character image until the model converges; the training object attribute features and the training virtual character attribute features are the attribute features for the real object and the attribute features for the virtual character extracted in the training stage respectively; The method for generating the virtual character image is as follows: obtaining a training set of facial images with different characteristics; inputting the training sets of facial images with different characteristics into different generators for training; after completing the training, based on the network parameters of at least one layer in the first target generator, replacing the network parameters of the corresponding layer in the second target generator to obtain a hybrid generator; the first target generator and the second target generator both belong to the trained generator; and generating the virtual character image through the hybrid generator.

14. The device according to claim 13, characterized in that The extraction module is also used to extract target video frames containing facial features of the target real object from the user video to be released or the film and television video to be played; wherein, the target video frames extracted from different user videos or different film and television videos belong to the facial images of different target real objects.

15. The device according to claim 14, characterized in that The decoding module is also used to input the object attribute features into the dedicated decoder after the universal encoder extracts the object attribute features, so that the dedicated decoder decodes the posture features and expression features in each of the object attribute features in turn based on the virtual character attribute features learned during the training process.

16. A face replacement model processing device, characterized in that: The device comprises: A first extraction module is used to extract identity features of a real object from a face image sample; A second extraction module is used to extract attribute features from the face image samples and the virtual character images in sequence through a universal encoder to obtain training object attribute features and training virtual character attribute features, respectively; the training object attribute features and the training virtual character attribute features are attribute features for the real object and attribute features for the virtual character extracted in the training phase, respectively; A decoding module, configured to decode the identity feature and the training object attribute feature by a universal decoder to obtain a first composite image; and decode the training virtual character attribute feature by a dedicated decoder to obtain a second composite image; an adjustment module, configured to adjust parameters of the universal encoder and the universal decoder according to a difference value between the first synthetic image and the human face image sample; and to adjust parameters of the universal encoder and the dedicated decoder according to a loss value between the second synthetic image and the virtual character image; A construction module is used to form a face replacement model based on the universal encoder and the dedicated decoder when the model converges; the face replacement model is used to perform face replacement on face images of different target real objects; An acquisition module is used to obtain a training set of face images with different characteristics; An input module, used to input the face image training sets with different characteristics into different generators for training; A replacement module, used for replacing the network parameters of the corresponding layer in the second target generator based on the network parameters of at least one layer in the first target generator after the training is completed, so as to obtain a hybrid generator; the first target generator and the second target generator both belong to the generator after training; A generation module is used to generate the virtual character image through the hybrid generator.

17. The device according to claim 16, characterized in that The first extraction module is also used to extract identity features from face image samples through a face recognition model to obtain identity features of a real object; the identity features include iris features, nose features and lip features.

18. The device according to claim 16, characterized in that The first target generator is used to generate an image with a first characteristic; the second target generator is used to generate an image with a second characteristic; The generation module is also used to obtain the image to be converted; input the image to be converted into the hybrid generator, so that the network layer in the hybrid generator performs image processing on the image to be converted to obtain a virtual character image having the first characteristic and the second characteristic.

19. The device according to claim 18, characterized in that The generation module is further used to obtain at least two images to be converted that are collected under different illuminations; the real object in each of the images to be converted has different postures and expressions; The generating module is further used to sequentially perform image processing on at least two of the to-be-converted images to obtain at least two virtual character images having the first characteristic and the second characteristic; The virtual character in each of the virtual character images has different postures and expressions.

20. The device according to claim 18, characterized in that The first characteristic and the second characteristic are different ages, or different skin colors, or different dressing styles; or, The first feature and the second feature are combinations of age, skin color and dressing style.

21. The device according to claim 16, characterized in that The replacement module is also used to determine the network layer of the parameters to be replaced in the second target generator; determine the network layer that matches the network layer of the parameters to be replaced in the first target generator; and replace the network parameters of the network layer of the parameters to be replaced based on the network parameters of the matching network layer in the first target generator.

22. The device according to claim 16, characterized in that The universal encoder includes a first universal encoder and a second universal encoder that share network parameters; The second extraction module is also used to normalize the facial image samples and the virtual character images respectively; input the processed facial image samples into the first universal encoder so that the first universal encoder extracts attribute features from the facial image samples; and extract attribute features from the processed virtual character images through the second universal encoder.

23. The device according to any one of claims 16 to 22, characterized in that The difference value between the first synthetic image and the face image sample includes a first difference value and a second difference value; The adjustment module is also used to determine a first difference value based on the first synthetic image and the face image sample; extract image features from the first synthetic image and the face image sample respectively to obtain first synthetic image features and face image features; determine a second difference value based on the first synthetic image features and the face image features; and adjust parameters of the universal encoder and the universal decoder based on the first difference value and the second difference value.

24. The device according to any one of claims 16 to 22, characterized in that The loss value between the second synthetic image and the virtual character image includes a first loss value, a second loss value and a third loss value; The adjustment module is further used to determine a first loss value according to the second synthetic image and the virtual character image; extract image features from the second synthetic image and the virtual character image respectively to obtain second synthetic image features and virtual character image features; determine a second loss value according to the second synthetic image features and the virtual character image features; and discriminate the second synthetic image and the virtual character image respectively through a discriminator to obtain a first discrimination result and a second discrimination result; determining a third loss value according to the first discrimination result and the second discrimination result; Parameters of the universal encoder and the dedicated decoder are adjusted according to the first loss value, the second loss value and the third loss value.

25. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

26. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

27. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images