Image processing method, model training method and related devices

Abstract

The embodiment of the invention discloses an image processing method, a model training method and a related device, at least relates to technologies such as artificial intelligence and the like, can be applied to scenes such as cloud technology, artificial intelligence, intelligent traffic, aided driving and big data, and is used for paying attention to details such as five sense organs in an image and improving the image generation quality and the face replacement effect. The image processing method comprises the following steps: acquiring a first face image and a second face image; replacing the face region of the first face image with the face region of the second face image based on a target generation model to obtain a target face image, the target generation model is a model obtained by training the initial generation model by taking a source sample image, a target sample image, a label sample image, a first local five-sense-organ image of the target sample image and a second local five-sense-organ image of the label sample image as training data; the target sample image is obtained by replacing the face area of the source sample image with the face area of the template sample image by the initial generation model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a method for image processing, a method for model training, and related devices. Background Art

[0002] The replacement of facial images has a wide range of application scenarios, such as film and television portrait production, game character design, virtual image creation, privacy protection, and other scenarios. Taking the film and television production scenario as an example, when an actor is unable to complete a professional action, a professional can complete it first, and later use the facial image replacement method to replace the face of the professional with the face of the actor to complete more targeted film and television production on the premise of ensuring safety.

[0003] In the traditional facial image replacement scheme, training data is usually constructed by using a source image, a template image, and a label image, and a generation network structure is completed and trained from the overall image difference. In this way, after the generation network structure is trained, the generation network structure is used to complete the replacement of the facial area of the source image to the facial area of the template image to complete the overall replacement processing of the facial image. However, this traditional replacement method needs to optimize the entire facial area, resulting in poor generation quality of the finally replaced image, which greatly affects the facial replacement effect. Summary of the Invention

[0004] The embodiments of the present application provide a method for image processing, a method for model training, and related devices, which are used to focus on local details of facial features in an image, improve the generation quality of a target facial image, and thus enhance the facial replacement effect.

[0005] In a first aspect, the embodiments of the present application provide a method for image processing. The method includes: obtaining a first facial image and a second facial image, where the first facial image and the second facial image are different; replacing the facial area of the first facial image with the facial area of the second facial image based on a target generation model to obtain a target facial image, where the target generation model is a machine learning model obtained by training an initial generation model with a source sample image, a target sample image, a label sample image, a first local facial feature map of the target sample image, and a second local facial feature map of the label sample image, and the target sample image is an image obtained by replacing the facial area in the source sample image with the facial area in a template sample image by the initial generation model.

[0006] Second aspect, an embodiment of the present application provides a method for model training. The method includes: obtaining a source sample image, a template sample image, and a label sample image, where the source sample image is different from the template sample image; replacing the facial region in the source sample image with the facial region in the template sample image based on an initial generation model to generate a target sample image; extracting a first local facial feature map of the target sample image and a second local facial feature map of the label sample image; updating the model parameters of the initial generation model based on the first local facial feature map, the second local facial feature map, the target sample image, the label sample image, and the source sample image to obtain a target generation model, where the target generation model is used to replace the facial region of a first facial image with the facial region of a second facial image to obtain a target facial image, and the first facial image is different from the second facial image.

[0007] Third aspect, an embodiment of the present application provides an image processing apparatus. The image processing apparatus includes an acquisition unit and a processing unit. The acquisition unit is configured to obtain a first facial image and a second facial image, where the first facial image is different from the second facial image. The processing unit is configured to replace the facial region of the first facial image with the facial region of the second facial image based on a target generation model to obtain a target facial image, where the target generation model is a machine learning model obtained by training the initial generation model with the source sample image, the target sample image, the label sample image, the first local facial feature map of the target sample image, and the second local facial feature map of the label sample image as training data, and the target sample image is an image obtained by replacing the facial region in the source sample image with the facial region in the template sample image by the initial generation model.

[0008] In some alternative embodiments, the processing unit is further configured to extract the identity feature of the first facial image before replacing the facial region of the first facial image with the facial region of the second facial image based on the target generation model to obtain a target facial image. The processing unit is configured to: perform encoding processing on the first facial image and the second facial image based on the target generation model to obtain an encoded feature vector; perform feature fusion on the encoded feature vector and the identity feature of the first facial image to obtain a fused feature vector; perform decoding processing on the fused feature vector based on the target generation model to generate a target facial image.

[0009] In some other alternative embodiments, a processing unit is configured to: calculate the feature mean of the encoded feature vector and the feature standard deviation of the encoded feature vector; calculate the feature mean of the identity feature of the first facial image and the feature standard deviation of the identity feature; and determine a fused feature vector based on the encoded feature vector, the feature mean of the encoded feature vector, the feature standard deviation of the encoded feature vector, the feature mean of the identity feature of the first facial image, and the feature standard deviation of the identity feature.

[0010] In a fourth aspect, an embodiment of the present application provides a model training device. The model training device includes an acquisition module and a processing module. The acquisition module is configured to acquire a source sample image, a template sample image, and a label sample image, where the source sample image is different from the template sample image. The processing module is configured to replace the facial region in the source sample image with the facial region in the template sample image based on an initial generation model to generate a target sample image. The processing module is configured to extract a first local facial feature map of the target sample image and a second local facial feature map of the label sample image. The processing module is configured to update the model parameters of the initial generation model based on the first local facial feature map, the second local facial feature map, the target sample image, the label sample image, and the source sample image to obtain a target generation model, where the target generation model is configured to replace the facial region of a first facial image with the facial region of a second facial image to obtain a target facial image, and the first facial image is different from the second facial image.

[0011] In some alternative embodiments, the processing module is further configured to extract the identity feature of the source sample image before replacing the facial region in the source sample image with the facial region in the template sample image based on the initial generation model to generate a target sample image. A processing unit is configured to: perform encoding processing on the facial region in the source sample image and the facial region in the template sample image based on an encoding module in the initial generation model to obtain a first feature vector; perform feature fusion on the first feature vector and the identity feature to obtain a second feature vector; and perform decoding processing on the second feature vector based on a decoding module in the initial generation model to generate a target sample image.

[0012] In some other alternative embodiments, the processing module is configured to: calculate the image difference between the first partial facial feature map and the second partial facial feature map to obtain a first loss value; calculate the image difference between the target sample image and the label sample image to obtain a second loss value; calculate the image difference between the target sample image and the source sample image to obtain a third loss value; update the model parameters of the initial generation model based on the first loss value, the second loss value, and the third loss value to obtain a target generation model.

[0013] In some other alternative embodiments, the processing module is configured to: calculate the pixel difference between the first partial facial feature map and the second partial facial feature map based on the pixel information of the first partial facial feature map and the pixel information of the second partial facial feature map to obtain a first pixel difference value; extract multiple facial features of the first partial facial feature map and extract multiple facial features of the second partial facial feature map, and calculate the facial feature difference value between the multiple facial features of the first partial facial feature map and the multiple facial features of the second partial facial feature map; perform a discrimination process on the first partial facial feature map based on a preset discrimination model to obtain a discrimination result of the first partial facial feature map; calculate a result loss value of the first partial facial feature map based on the discrimination result of the first partial facial feature map; calculate the sum of the first pixel difference value, the facial feature difference value, and the result loss value of the first partial facial feature map to obtain a first loss value.

[0014] In some other alternative embodiments, the processing module is configured to: calculate the absolute feature difference between each facial feature in the first partial facial feature map and the corresponding facial feature in the second partial facial feature map to obtain a plurality of absolute feature differences; calculate the sum of the plurality of absolute feature differences to obtain the facial feature difference value between the multiple facial features of the first partial facial feature map and the multiple facial features of the second partial facial feature map.

[0015] In some other alternative embodiments, the processing module is configured to: perform feature extraction processing on the first partial facial feature map based on a preset feature extraction model to obtain multiple facial features of the first partial facial feature map; perform feature extraction processing on the second partial facial feature map based on the preset feature extraction model to obtain multiple facial features of the second partial facial feature map.

[0016] In some other alternative embodiments, the processing module is configured to: calculate a pixel difference between the target sample image and the labeled sample image based on the pixel information of the target sample image and the pixel information of the labeled sample image, so as to obtain a second pixel difference value; extract multiple image features of the target sample image, and extract multiple image features of the labeled sample image, and calculate a feature difference value between the multiple image features of the target sample image and the multiple image features of the labeled sample image; perform a discrimination process on the target sample image based on a preset discrimination model to obtain a discrimination result of the target sample image; calculate a result loss value of the target sample image based on the discrimination result of the target sample image; calculate the sum of the second pixel difference value, the feature difference value, and the result loss value of the target sample image to obtain a second loss value.

[0017] In some other alternative embodiments, the processing module is configured to: calculate an absolute feature difference value between each of the multiple image features of the target sample image and the corresponding image feature in the labeled sample image to obtain multiple absolute image feature difference values; calculate the sum of the multiple absolute image feature difference values, and calculate a feature difference value between the multiple image features of the target sample image and the multiple image features of the labeled sample image.

[0018] In some other alternative embodiments, the processing module is configured to: extract the identity feature of the target sample image and the identity feature of the source sample image; calculate a similarity distance between the identity feature of the target sample image and the identity feature of the source sample image to obtain a similarity degree between the identity feature of the target sample image and the identity feature of the source sample image; obtain a third loss value based on the similarity degree.

[0019] In some other alternative embodiments, the processing module is further configured to: perform a discrimination process on the second local facial feature map, the target sample image, and the labeled sample image respectively based on the preset discrimination model to obtain a discrimination result of the second local facial feature map, a discrimination result of the target sample image, and a discrimination result of the labeled sample image; calculate a facial feature discrimination result loss value based on the discrimination result of the first local facial feature map and the discrimination result of the second local facial feature map, and calculate an image discrimination result loss value based on the discrimination result of the target sample image and the discrimination result of the labeled sample image; update the model parameters of the preset discrimination model based on the facial feature discrimination result loss value and the image discrimination result loss value.

[0020] In some other alternative embodiments, a processing module is configured to: perform facial feature region segmentation processing on the target sample image based on a preset segmentation model to obtain facial feature key points in the target sample image, and obtain a first local facial feature map based on the facial feature key points in the target sample image; perform facial feature region segmentation processing on the label sample image based on the preset segmentation model to obtain facial feature key points in the source sample image, and obtain a second local facial feature map based on the facial feature key points in the label sample image.

[0021] In a fifth aspect of the embodiments of the present application, an image processing device is provided, including: a memory, an input / output (I / O) interface, and a memory. The memory is used to store program instructions. The processor is configured to execute the program instructions in the memory to execute the image processing method corresponding to the embodiments of the first aspect above; or, execute the model training method corresponding to the embodiments of the second aspect above.

[0022] In a sixth aspect of the embodiments of the present application, a computer-readable storage medium is provided. Instructions are stored in the computer-readable storage medium, and when executed on a computer, cause the computer to execute the image processing method corresponding to the embodiments of the first aspect above; or, execute the model training method corresponding to the embodiments of the second aspect above.

[0023] In a seventh aspect of the embodiments of the present application, a computer program product including instructions is provided. When executed on a computer or a processor, cause the computer or the processor to execute the image processing method corresponding to the embodiments of the first aspect above; or, execute the model training method corresponding to the embodiments of the second aspect above.

[0024] From the above technical solutions, it can be seen that the embodiments of the present application have the following advantages:

[0025] In the embodiments of the present application, the source sample image, the target sample image, the label sample image, the first partial facial feature map of the target sample image, and the second partial facial feature map of the label sample image are used as training data. Then, based on this training data, the initial generation model is trained to obtain the target generation model. The described target sample image is an image obtained by replacing the facial region in the source sample image with the facial region in the template sample image based on the initial generation model. In this way, after obtaining the first facial image and the second facial image, the first facial image and the second facial image are used as the input of the target generation model, so that the facial region of the first facial image can be replaced with the facial region of the second facial image through the target generation model, thereby obtaining the target facial image. Through the above method, in the process of training the target generation model, not only the source sample image, the label sample image, and the target sample image obtained based on the source sample image and the template sample image are considered, but also the first partial facial feature map of the target sample image and the second partial facial feature map of the label sample image are taken into account, so that the generated target generation model can not only focus on and optimize the overall facial image, but also focus on the local details of the facial features in the image, thereby completing the detail optimization process of the local details of the facial features. In this way, in the subsequent process of using the target generation model to process the first facial image and the second facial image, the generation quality of the generated target facial image can be greatly improved, and the facial replacement effect can be enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0027] Figure 1 FIG. shows a schematic diagram of an application scenario of facial replacement provided by an embodiment of the present application;

[0028] Figure 2 FIG. shows an alternative schematic diagram of the system framework provided by an embodiment of the present application;

[0029] Figure 3 FIG. shows a schematic flowchart of the method for model training provided by an embodiment of the present application;

[0030] Figure 4 FIG. shows an alternative structural schematic diagram of the initial generation model provided by an embodiment of the present application;

[0031] Figure 5 FIG. shows a schematic flowchart of updating model parameters provided by an embodiment of the present application;

[0032] Figure 6 Schematic flowchart of the image processing method provided by the embodiment of the present application shown;

[0033] Figure 7 Schematic diagram of the functional modules of the image processing device provided by the embodiment of the present application shown;

[0034] Figure 8 Schematic diagram of the functional modules of the model training device provided by the embodiment of the present application shown;

[0035] Figure 9 Schematic diagram of the hardware structure of the image processing device provided by the embodiment of the present application shown. Detailed implementation manners

[0036] The embodiment of the present application provides an image processing method, a model training method and related devices, which are used to focus on local details of facial features in an image, etc., improve the generation quality of the target facial image, and further improve the facial replacement effect.

[0037] It can be understood that in the specific implementation manners of the present application, data related to user information, etc. are involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0038] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0039] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above accompanying drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0040] In recent years, face-changing based on videos or images has been widely used in different application scenarios. For example, it can be used in scenarios such as film and television portrait production, game character design, virtual images, privacy protection, etc., which are not limited in this application. For example, in the scene of film and television production, when the actors cannot complete professional actions, professionals can complete them first, and then the actors can be automatically replaced by face-changing; or it can be used to replace bad artists to avoid reshooting, which can save a lot of costs. Alternatively, in the scene of virtual images, users can use virtual characters to change faces, increase the fun of live broadcasts, and even protect personal privacy.

[0041] Face-changing methods are sometimes referred to as face replacement. The source image and the template image are used as inputs to a generative model to output a face replacement image. However, in traditional face replacement methods, the optimization process is mainly based on the overall facial area in the image, and it is impossible to pay more attention to the details of the facial features, resulting in poor quality of the image generated by the replacement, which greatly affects the effect of face replacement.

[0042] Therefore, in order to solve the above-mentioned technical problems, the embodiment of the present application provides a method for image processing. Correspondingly, the embodiment of the present application also provides a method for model training corresponding to the method for image processing. Through the method for model training provided by the present application, a target generation model that considers both the overall facial image and local details such as facial features can be trained. That is to say, in the process of training the target generation model, the source sample image, the label sample image, and the target sample image obtained based on the source sample image and the template sample image are considered, and the first local facial features map of the target sample image and the second local facial features map of the label sample image need to be taken into consideration, so that the generated target generation model can not only pay attention to and optimize the overall facial image, but also focus on the local details of the facial features in the image, thereby completing the detail optimization processing of the local details of the facial features. In this way, in the process of the subsequent model use stage, the target generation model generated by the model training stage can pay attention to the detailed features such as the facial features of the input first facial image and the second facial image, so that the generation quality of the generated target facial image can be greatly improved, and the facial replacement effect can be improved.

[0043] With the research and progress of artificial intelligence (AI) technology, AI technology has been studied and applied in multiple fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, driverless, autonomous driving, drones, robots, smart healthcare, smart customer service, etc. It is believed that with the development of technology, AI technology will be applied in more fields and play an increasingly important role. Therefore, the image processing method and model training method provided in the embodiments of this application can both be implemented based on artificial intelligence.

[0044] Artificial intelligence technology is a comprehensive discipline with a wide range of fields involved, including both hardware-level and software-level technologies. Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technologies mainly include several major directions such as computer vision technology, speech technology, natural language processing technology, and machine learning / deep learning.

[0045] In the embodiments of this application, the artificial intelligence technologies mainly involved include the above-mentioned computational vision (image), machine learning / deep learning, etc. For example, it can involve video processing, video semantic understanding (VSU), face recognition, etc. in computer vision (CV). Target recognition, target detection and localization, etc. are included in video semantic understanding; face 3D reconstruction, face detection, etc. are included in face recognition.

[0046] The image processing method provided in this application can be applied to an image processing device with data processing capabilities. Exemplarily, the model training method provided in this application can also be applied to the above-mentioned image processing device. For example, the described image processing device may include, but is not limited to, a terminal device, or a server, etc. Among them, the terminal device may include, but is not limited to, a smart phone, a desktop computer, a laptop computer, a tablet computer, a smart speaker, a vehicle-mounted device, a smart watch, a wearable smart device, a smart voice interaction device, a smart home appliance, an aircraft, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms. This application does not make specific limitations. In addition, the mentioned terminal device and server can be directly or indirectly connected through wired communication or wireless communication, etc. This application does not make specific limitations.

[0047] The above-mentioned image processing device can be capable of implementing the above-mentioned computer vision technology. As a scientific discipline, the described computer vision technology studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. The large model technology has brought important changes to the development of computer vision technology. Pretrained models in the visual field such as swin-transformer, ViT, V-MOE, MAE, etc. can be quickly and widely applied to downstream specific tasks after fine-tuning. Computer vision technology usually includes technologies such as image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, etc., and also includes common biometric recognition technologies such as face recognition and fingerprint recognition.

[0048] In this application, the recognition and processing of objects such as videos or images can be achieved by means of technologies such as image processing and video processing in computer vision technology. In the embodiments of this application, the image processing device can, by implementing the above computer vision technology, replace the facial region in the source sample image with the facial region in the template sample image through the initial generation model to obtain the target sample image, and use the source sample image, the target sample image, the labeled sample image, the first partial facial features map of the target sample image, and the second partial facial features map of the labeled sample image as training data to train the initial generation model to obtain a machine learning model. Exemplarily, the image processing device in this application can, by implementing the above computer vision technology, also achieve functions such as replacing the facial region of the first facial image with the facial region of the second facial image based on the target generation model to obtain the target facial image.

[0049] In addition, the image processing device can also have machine learning capabilities. Machine learning and deep learning generally include technologies such as neural networks. The artificial intelligence model adopted in the method for image processing and the method for model training provided in the embodiments of this application mainly involves the application of neural networks, and the generation of the target facial image or the target sample image is achieved through the neural network, etc. Specifically, it is not limited in this application.

[0050] Exemplarily, Figure 1 shows a schematic diagram of an application scenario of facial replacement provided by an embodiment of this application. As Figure 1 shown, in this application scenario, it includes face image A and face image B. At this time, the facial region in face image A can be replaced with the facial region in face image B to generate face image C. For this face image C, it can be understood as the picture obtained by replacing the identity in face image B with face image A. Or rather, in this face image C, it is necessary to maintain information such as the facial expression, angle, or background in face image B, and also be as similar as possible to the facial features and other details in face image A.

[0051] Figure 2 shows an optional schematic diagram of the system framework provided by an embodiment of this application. As Figure 2 shown, in this system framework, it includes a model training stage and a model usage stage.

[0052] In the model training stage, it is necessary to obtain the source sample image, the template sample image, and the labeled sample image. And the source sample image is different from the template sample image. The facial region in the source sample image is replaced with the facial region in the template sample image through the initial generation model to generate the target sample image. In this target sample image, it is necessary to maintain information such as the facial expression, angle, or background in the template sample image, and also be as similar as possible to the facial features and other details in the source sample image. For example, the aboveFigure 1 The face image A shown in Figure 1 can be understood as the source sample image described herein, and the face image B shown in Figure 1 can be understood as the template sample image described herein. Correspondingly,

[0053] In this way, the first local facial feature map of the target sample image is extracted from the target sample image, and the second local facial feature map of the label sample image is extracted from the label sample image. Thus, the first local facial feature map, the second local facial feature map, the target sample image, the label sample image, and the source sample image are used as training data to complete the model update process of the initial generation model, thereby obtaining the target generation model after updating. Optionally, the image difference between the first local facial feature map and the second local facial feature map, the image difference between the target sample image and the label sample image, and the image difference between the target sample image and the source sample image can be calculated, so as to update the model parameters of the initial generation model based on the differences and train to obtain the target generation model.

[0054] In this way, in the model usage stage, the first facial image and the second facial image are first obtained, and then the target generation model trained in the above model training stage is used to process the first facial image and the second facial image to complete the replacement of the facial region of the first facial image with the facial region of the second facial image, thereby obtaining the target facial image.

[0055] In addition, from Figure 2 it can also be seen that in the initial generation model, an encoding module and a decoding module are included. Exemplarily, in the process of generating the target sample image through the initial generation model, the source sample image and the template sample image can be encoded based on the encoding module in the initial generation model, and then the encoded feature vectors can be decoded using the decoding module, thereby obtaining the target sample image. The more detailed implementation process can be understood by referring to the content shown in the subsequent Figure 4 and will not be elaborated here.

[0056] Exemplarily, the model training method and the image processing method provided in this application can also be applied to scenarios such as cloud technology, artificial intelligence, intelligent transportation, assisted driving, big data, etc., and the specific application is not limited in this application.

[0057] As a schematic description, since the execution of the image processing method described above requires the use of the target processing model trained by the model training method in the early stage. Therefore, the model training method provided in the embodiments of this application will be described in detail from the perspective of embodiments below. Exemplarily, Figure 3The flowchart shows the method for model training provided by the embodiments of the present application. As Figure 3 shown, the method for model training at least includes the following steps:

[0058] 301. Obtain a source sample image, a template sample image, and a label sample image, where the source sample image is different from the template sample image.

[0059] In this example, the source sample image can be understood as the image that needs to perform a face replacement operation. For example, it can be understood with reference to the face image A mentioned above Figure 1 . Similarly, the template sample image can be understood as the template image for providing face replacement. For example, it can be understood with reference to the face image B mentioned above Figure 1 . Through the face template in the template sample image, it is possible to perform face replacement on the face in the source sample image according to the face template.

[0060] Regarding the source sample image and the template sample image mentioned above, these two images are not the same. More specifically, the face in the source sample image is different from the face in the template sample image. For example, taking the face image A and the face image B shown above Figure 1 as an example, the face in the face image A is different from the face in the face image B.

[0061] Optionally, the source sample image and the template sample image mentioned above include but are not limited to face images, such as face images of humans, face images of animals, face images of cartoon characters, or face images of other virtual humans, etc. Specifically, it is not limited in this application. In the embodiments of this application, only face images are taken as examples for illustration.

[0062] Exemplarily, regarding how to obtain the source sample image, its acquisition method is not limited in this application. For example, it can be obtained from the face images stored in the database. Or, it can also be obtained by taking a face photo of an object, or by intercepting the face image of an object from a video. Or, it can also be obtained by first obtaining the source image of an object. It should be noted that the difference between the source image and the source sample image is that the source sample image can be understood as the face image in the face area of the source image. After obtaining the source image, face detection can be performed on the source image by using face detection and other methods to obtain the face area in the source image; further, face alignment is performed on the face area of the source image to obtain multiple facial key points, such as eye key points, mouth corner key points, etc. In this way, the source sample image can be obtained by intercepting the source image according to multiple facial key points.

[0063] Similarly, regarding how to obtain the template sample image, it can also be understood with reference to the above method for obtaining the source sample image, and details are not described here.

[0064] Exemplarily, in addition to obtaining the source sample image and the template sample image, in order to determine whether the image generated after subsequent face replacement is accurate, the present application also needs to obtain a label sample image during the model training process. For example, the label sample image can be obtained by means of manual annotation. Through this label sample image, the true label of the generated target sample image can be truly reflected.

[0065] 302. Replace the face region in the source sample image with the face region in the template sample image based on the initial generation model to generate a target sample image.

[0066] In this example, the initial generation model may include, but is not limited to, a generation network in a generative adversarial network (GAN), etc., and is not specifically limited in the present application. After obtaining the source sample image and the template sample image, the source sample image and the template sample image are used as the input of the initial generation model, so as to replace the face region in the source sample image with the face region in the template sample image through the initial generation model, thereby generating a target sample image.

[0067] Exemplarily, since it is desired that the face in the generated target sample image is as close as possible to the face in the source sample image, the identity feature of the source sample image can also be used for constraint. As a schematic description, before performing step 302 to replace the face region, the identity feature of the source sample image can be extracted, for example, through a model such as a face recognition network. Through the identity feature of the source sample image, the identity of the face in the source sample image can be determined. As a schematic description, the feature dimension of the identity feature of the source sample image is generally 1024 dimensions, and is not specifically limited in the present application. In addition, in the present application, for example, the identity feature of the source sample image can be represented by the symbol source_id_features, and no specific symbol definition is given.

[0068] Moreover, from the foregoing Figure 2 it can be seen that the initial generation model at least includes an encoding module and a decoding module. Exemplarily, Figure 4 shows an optional structural schematic diagram of the initial generation model provided by an embodiment of the present application. As Figure 4 shown, in this initial generation model, the encoding module includes a plurality of convolutional layers (conv), and the decoding module includes a plurality of deconvolutional layers (deconv).

[0069] Based on this, based on Figure 4 the shown initial generation model, the face region in the source sample image can be replaced with the face region in the template sample image, and the process can be understood with reference to the following manner, that is:

[0070] First, based on the encoding module in the initial generation model, encode the facial regions in the source sample image and the facial regions in the template sample image to obtain a first feature vector. For example, the source sample image and the template sample image can be stitched or overlapped, and then the stitched or overlapped image is successively encoded by multiple convolutional layers in the encoding module to obtain the first feature vector.

[0071] For example, assume that the image size of the source sample image is 256×256×3, and the image size of the template sample image is 256×256×3. At this time, the size of the stitched image becomes 512×512×6. In this way, the stitched image with a size of 512×512×6 is used as the input of the encoding module, and is gradually encoded by multiple convolutional layers into 256×256×32, 128×128×64, 64×64×128, 32×32×256, so as to obtain an intermediate vector in the latent space, that is, the first feature vector. For example, in this application, the symbol swap_features can be used to represent the first feature vector, and the specific symbols in this application are not limited.

[0072] After obtaining the first feature vector and the identity feature, perform feature fusion on the first feature vector and the identity feature to obtain a second feature vector. As a schematic description, the feature mean and feature standard deviation of the first feature vector can be calculated based on the first feature vector, and the feature mean and feature standard deviation of the identity feature of the source sample image can be calculated based on the identity feature of the source sample image; subsequently, based on the first feature vector, the feature mean and feature standard deviation of the first feature vector, and the feature mean and feature standard deviation of the identity feature of the source sample image, calculate the second feature vector. For example, the second feature vector satisfies where x represents swap_features, that is, the first feature vector mentioned above; y represents source_id_features, that is, the identity feature mentioned above; μ(x) represents the feature mean of the first feature vector, σ(x) represents the feature standard deviation of the first feature vector, μ(y) represents the feature mean of the identity feature; σ(y) represents the feature standard deviation of the identity feature; AdaIN(x,y) represents the second feature vector, where AdaIN() represents the AdaIN algorithm.

[0073] In this way, after obtaining the second feature vector, use the second feature vector as the input of the decoding module in the initial generation model, and perform decoding processing on the second feature vector through the decoding module in the initial generation model, for example, perform deconvolution operations to generate the target sample image.

[0074] For example, the second feature vector is gradually subjected to deconvolution calculations through multiple deconvolution layers in the decoding module, and is gradually decoded into 32×32×256, 64×64×128, 128×128×64, 256×256×32, 512×512×3, so as to obtain the target sample image.

[0075] It should be noted that the encoding module mentioned in this application is sometimes also referred to as an encoder or an encoding model, and the name is not specifically limited in this application. Similarly, the decoding module mentioned in this application is sometimes also referred to as a decoder or a decoding model, and the name is not specifically limited in this application. In addition, for the multiple convolutional layers and multiple deconvolution layers shown above Figure 4 the number of their layers can be determined according to requirements and is not specifically limited in this application.

[0076] 303. Extract the first local facial features map of the target sample image and the second local facial features map of the label sample image.

[0077] In this example, during the face replacement process, the generation quality of the generated image, especially the generation quality of the facial features in the image, will affect whether the generated image can be applied to different scenarios. Moreover, for a facial image, users often pay special attention to the details of the facial features, such as whether the lines of the double eyelids are smooth, the naturalness of the eye sockets, the degree of gaze of the line of sight, the details of the corners of the mouth, and so on.

[0078] Based on this, after generating the target sample image, the first local facial features map of the target sample image can also be extracted. Through this first local facial features map, the facial features of the face in the target sample image can be reflected. As a schematic description, specifically, the target sample image can be used as the input of a preset segmentation model to perform facial features region segmentation processing on the target sample image through the preset segmentation model, so as to extract the facial features key points in the target sample image, such as eye key points, mouth corner key points, etc. Subsequently, the first local facial features map is generated based on the facial features key points in the target sample image. For example, in this application, the first local facial features map satisfies eye_mouth_fake = fake × eye_mouth_mask. Wherein, eye_mouth_mask represents the pixel value of the facial features key points, fake represents the pixel value of the target sample image, and eye_mouth_fake represents the pixel value of the first local facial features map of the target sample image.

[0079] In order to more precisely determine whether the first partial facial feature map is accurate in the subsequent stage, so as to improve the learning ability of the generation model. Similarly, it is also necessary to extract the second partial facial feature map of the label sample image. Through this second partial facial feature map, the facial features of the face in the label sample image can be reflected. As a schematic description, specifically, the label sample image can be used as the input of a preset segmentation model to perform facial feature region segmentation processing on the label sample image through the preset segmentation model, so as to extract the facial feature key points in the label sample image, such as eye key points, mouth corner key points, etc. Subsequently, the second partial facial feature map is generated based on the facial feature key points in the label sample image. For example, in this application, the second partial facial feature map satisfies eye_mouth_GT = GT × eye_mouth_mask. Wherein, eye_mouth_mask represents the pixel value of the facial feature key points, GT represents the pixel value of the label sample image, and eye_mouth_GT represents the pixel value of the second partial facial feature map of the target sample image.

[0080] 304. Update the model parameters of the initial generation model based on the first partial facial feature map, the second partial facial feature map, the target sample image, the label sample image, and the source sample image to obtain a target generation model, which is used to replace the facial region of the first facial image with the facial region of the second facial image to obtain a target facial image, and the first facial image is different from the second facial image.

[0081] In this example, after generating the target sample image and extracting the first partial facial feature map and the second partial facial feature map, combined with the label sample image and the source sample image, they are jointly used as training data to perform parameter adjustment processing on the model parameters of the initial generation model, so as to train and obtain the target generation model. That is to say, the target generation model is a machine learning model obtained by training the initial generation model with the target sample image, the first partial facial feature map, the second partial facial feature map, the label sample image, and the source sample image as training data.

[0082] As a schematic description, since it is desired that the output of the deep neural network be as close as possible to the truly desired predicted value, the weight vector of each layer of the neural network can be updated according to the difference between the current network's predicted value and the truly desired target value (of course, there is usually an initialization process before the first update, that is, parameters are pre-configured for each layer in the deep neural network). For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict lower, and continuous adjustment is made until the neural network can predict the truly desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function, and they are important equations for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0083] Therefore, during the specific training process, the loss function can be added synchronously to improve the learning ability of the generation model. Exemplarily, the differences between the first partial facial feature map and the second partial facial feature map, between the target sample image and the label sample image, and between the target sample image and the source sample image can be comprehensively considered to jointly construct the overall loss value, and then the model parameters of the initial generation model can be adjusted based on the overall loss value. As an alternative implementation manner, how to generate the overall loss value and adjust the model parameters of the initial generation model can be understood with reference to the Figure 5 flow schematic diagram shown below. As Figure 5 shown, this process at least includes steps 501 to 504, specifically as follows:

[0084] 501. Calculate the image difference between the first partial facial feature map and the second partial facial feature map to obtain the first loss value.

[0085] In this example, during the process of calculating the difference between the facial feature maps, the pixel difference perspective, the facial feature difference perspective, and the discrimination result perspective of the facial feature maps can also be comprehensively considered. It can be specifically understood with reference to the following method, that is:

[0086] In the case of pixel differences in the facial feature map, the pixel information of the first partial facial feature map can be obtained from the first partial facial feature map, such as eye_mouth_fake mentioned above. Similarly, the pixel information of the second partial facial feature map can also be obtained from the second partial facial feature map, such as eye_mouth_GT mentioned above. In this way, after obtaining the pixel information of the first partial facial feature map and the pixel information of the second partial facial feature map, based on the pixel information of the first partial facial feature map and the pixel information of the second partial facial feature map, calculate the pixel difference between the first partial facial feature map and the second partial facial feature map, so as to obtain the first pixel difference value.

[0087] For example, the first pixel difference value mentioned above satisfies eye_mouth_L1_Loss = abs(eye_mouth_fake – eye_mouth_GT). Where abs represents the absolute value operation, and eye_mouth_L1_Loss represents the first pixel difference value.

[0088] It should be noted that through this first pixel difference value, the pixel difference between the first partial facial feature map and the second partial facial feature map can be clearly known.

[0089] In consideration of the differences in facial features, multiple facial features of the first partial facial feature map and multiple facial features of the second partial facial feature map can also be extracted first. As a schematic description, a preset feature extraction model can be used to extract facial features. For example, the first partial facial feature map is used as the input of the preset feature extraction model to perform feature extraction processing on the first partial facial feature map through the preset feature extraction model to obtain multiple facial features of the first partial facial feature map. Similarly, the second partial facial feature map is used as the input of the preset feature extraction model to perform feature extraction processing on the second partial facial feature map through the preset feature extraction model to obtain multiple facial features of the second partial facial feature map. In this way, after extracting multiple facial features of the first partial facial feature map and multiple facial features of the second partial facial feature map, calculate the facial feature difference value between multiple facial features of the first partial facial feature map and multiple facial features of the second partial facial feature map. For example, the absolute difference in features between each facial feature in the first partial facial feature map and the corresponding facial feature in the second partial facial feature map can be calculated, and thus multiple absolute differences in features are obtained; further, sum these multiple absolute differences in features to obtain the facial feature difference value between multiple facial features of the first partial facial feature map and multiple facial features of the second partial facial feature map.

[0090] It should be noted that the preset feature extraction model mentioned above can include but is not limited to the AlexNet model, and specific limitations are not made in this application.

[0091] For example, taking the AlexNet model as the preset feature extraction model, 4 facial feature of the first partial facial features map can be extracted based on the AlexNet model. For example, eye_mouth_fake_F1, eye_mouth_fake_F2, eye_mouth_fake_F3, and eye_mouth_fake_F4 all satisfy AlexNet_fake_Feature(eye_mouth_fake).

[0092] Similarly, 4 facial features of the second partial facial features map can be extracted based on the AlexNet model. For example, eye_mouth_GT_F1, eye_mouth_GT_F2, eye_mouth_GT_F3, and eye_mouth_GT_F4 all satisfy AlexNet_GT_Feature(eye_mouth_GT).

[0093] In this way, the calculated facial feature difference value satisfies: eye_mouth_LPIPS_Loss = abs(eye_mouth_fake_F1 – eye_mouth_GT_F1) + abs(eye_mouth_fake_F2 – eye_mouth_GT_F2) + abs(eye_mouth_fake_F3 – eye_mouth_GT_F3) + abs(eye_mouth_fake_F4 - eye_mouth_GT_F4). Among them, abs(eye_mouth_fake_F1 – eye_mouth_GT_F1) represents the absolute feature difference between the facial feature F1 in the first partial facial features map and the facial feature F1 in the second partial facial features map. For other contents such as abs(eye_mouth_fake_F2 – eye_mouth_GT_F2), it can be understood by referring to the content of abs(eye_mouth_fake_F1 – eye_mouth_GT_F1), and will not be elaborated here.

[0094] In addition, considering the discrimination result of the facial feature map, the first partial facial feature map can be discriminated based on a preset discrimination model in the generative adversarial network to obtain the discrimination result of the first partial facial feature map, such as a discrimination probability value. Through this discrimination result, it can be known whether the first partial facial feature map is a real image or a fake image. In this way, the result loss value of the first partial facial feature map is calculated based on the discrimination result of the first partial facial feature map. For example, the result loss value of the first partial facial feature map satisfies: eye_mouth_G_Loss = log(1 – eye_mouth_D(fake)). Here, eye_mouth_G_Loss represents the result loss value of the first partial facial feature map, and eye_mouth_D(fake) represents the discrimination result of the first partial facial feature map.

[0095] In this way, after calculating the first pixel difference value (such as eye_mouth_L1_Loss mentioned above), the facial feature difference value (such as eye_mouth_LPIPS_Loss mentioned above), and the result loss value of the first partial facial feature map (such as eye_mouth_G_Loss mentioned above), sum the first pixel difference value, the facial feature difference value, and the result loss value of the first partial facial feature map, and use the obtained sum as the first loss value. For example, the first loss value Loss1 = eye_mouth_L1_Loss + eye_mouth_LPIPS_Loss + eye_mouth_G_Loss.

[0096] Optionally, weighted summation processing can also be performed on eye_mouth_L1_Loss, eye_mouth_LPIPS_Loss, and eye_mouth_G_Loss to obtain the first loss value after weighted summation. For example, the first loss value after weighted summation Loss1 = λ1 × eye_mouth_L1_Loss + λ2 × eye_mouth_LPIPS_Loss + λ3 × eye_mouth_G_Loss.

[0097] It should be noted that the described weight values λ1, λ2, and λ3 are adjustable coefficients, and the specific values are not limited in this application.

[0098] 502. Calculate the image difference between the target sample image and the label sample image to obtain the second loss value.

[0099] In this example, in addition to calculating the differences between the facial feature characteristics through the above step 501, the overall image difference can also be considered. During the process of calculating the overall loss value, the image difference between the target sample image and the label sample image can also be calculated. Exemplarily, during the process of calculating the difference between the overall images, it can also be comprehensively considered from the perspectives of pixel difference of the images, feature difference of the images, and discrimination result of the images. Specifically, it can be understood by referring to the following method, that is:

[0100] In the case of considering the pixel difference of the images, the pixel information of the target sample image can be obtained from the target sample image. Similarly, the pixel information of the label sample image can also be obtained from the label sample image. Further, based on the pixel information of the target sample image and the pixel information of the label sample image, the pixel difference between the target sample image and the label sample image is calculated, so as to obtain the second pixel difference value. Through this second pixel difference value, the pixel difference situation between the target sample image and the label sample image can be clearly known.

[0101] For example, the above-mentioned second pixel difference value satisfies L1_Loss = abs(fake – GT). Wherein, abs represents the absolute value operation, eye_mouth_L1_Loss represents the first pixel difference value, fake represents the pixel information of the target sample image, and GT represents the pixel information of the label sample image.

[0102] In the case of considering the image feature difference, multiple image features of the target sample image and multiple image features of the label sample image can be extracted first. As a schematic description, a preset feature extraction model can be used to extract the image features. For example, the target sample image is used as the input of the preset feature extraction model to perform feature extraction processing on the target sample image through the preset feature extraction model to obtain multiple image features of the target sample image. Similarly, the label sample image is used as the input of the preset feature extraction model to perform feature extraction processing on the label sample image through the preset feature extraction model to obtain multiple image features of the label sample image. In this way, after extracting multiple image features of the target sample image and multiple image features of the label sample image, the facial feature difference value between the multiple image features of the target sample image and the multiple image features of the label sample image is calculated. For example, the absolute feature difference between each image feature in the target sample image and the corresponding image feature in the label sample image can be calculated, and multiple absolute feature differences are calculated thereby; further, the sum of these multiple absolute feature differences is calculated to obtain the image difference value between the multiple image features of the target sample image and the multiple image features of the label sample image.

[0103] For example, taking the AlexNet model mentioned in the above step 501 as the preset feature extraction model,

[0104] Based on this AlexNet model, 4 image features of the target sample image can be extracted. For example, fake_F1, fake_F2, fake_F3, and fake_F4 all satisfy: AlexNet_fake_Feature(fake).

[0105] Similarly, based on this AlexNet model, 4 image features of the label sample image can be extracted. For example, GT_F1, GT_F2, GT_F3, and _GT_F4 all satisfy: AlexNet_GT_Feature(GT).

[0106] In this way, the calculated feature difference value satisfies: LPIPS_Loss = |fake_F1 – GT_F1| + |fake_F2 – GT_F2| + |fake_F3 – GT_F3| + |fake_F4 - GT_F4|. Among them, |fake_F1 – GT_F1| represents the absolute feature difference between the image feature F1 in the target sample image and the image feature F1 in the label sample image. For other contents such as |fake_F2 – GT_F2|, etc., it can be understood with reference to the content of |fake_F1 – GT_F1|, and no further elaboration will be made here.

[0107] Regarding the situation of the discrimination result of the image, the target sample image can also be discriminated first based on the preset discrimination model in the generative adversarial network to obtain the discrimination result of the target sample image, such as the discrimination probability value. Through the discrimination result of the target sample image, it can be known whether the target sample image is a real image or a fake image. In this way, the result loss value of the target sample image is calculated based on the discrimination result of the target sample image. For example, the result loss value of the target sample image satisfies: G_Loss = log(1 – D(fake)). Among them, G_Loss represents the result loss value of the target sample image, and D(fake) represents the discrimination result of the target sample image.

[0108] In this way, after calculating the second pixel difference value (such as the aforementioned L1_Loss), the feature difference value (such as the aforementioned LPIPS_Loss), and the result loss value of the target sample image (such as the aforementioned G_Loss), sum the second pixel difference value, the feature difference value, and the result loss value of the target sample image, and use the obtained sum as the second loss value. For example, the second loss value Loss2 = L1_Loss + LPIPS_Loss + G_Loss.

[0109] Optionally, the L1_Loss, LPIPS_Loss, and G_Loss can also be weighted and summed to obtain a second loss value after weighted summation. For example, the second loss value Loss2 after weighted summation = λ4 × L1_Loss + λ5 × LPIPS_Loss + λ6 × G_Loss

[0110] It should be noted that the described weights λ4, λ5, and λ6 are adjustable coefficients, and the specific values are not limited in this application.

[0111] 503. Calculate the image difference between the target sample image and the source sample image to obtain a third loss value.

[0112] In this example, since it is desired that the generated target sample image is as similar as possible to the source sample image, in the embodiments of this application, the image difference between the target sample image and the source sample image can also be calculated to obtain a third loss value.

[0113] As a schematic description, in the process of calculating the third loss value, the identity features of the target sample image and the identity features of the source sample image can be extracted respectively based on the face recognition network mentioned in the foregoing step 302. In this way, after the identity features of the target sample image and the identity features of the source sample image are extracted, calculate the similarity distance between the identity features of the target sample image and the identity features of the source sample image to obtain the similarity between the identity features of the target sample image and the identity features of the source sample image. For example, the above-mentioned similarity distance can include but is not limited to cosine similarity distance, Euclidean distance, etc., and is not specifically limited in this application. In this way, the third loss value is determined based on this similarity.

[0114] For example, taking the cosine similarity distance as an example, the above-mentioned third loss value satisfies ID_Loss = 1 - cosine_similarity(fake_id_features, source_id_features). Wherein, fake_id_features represents the identity features of the target sample image, source_id_features represents the identity features of the source sample image, and ID_Loss represents the third loss value.

[0115] In addition, cosine_similarity(fake_id_features, source_id_features) represents the cosine similarity distance between the identity features of the target sample image and the identity features of the source sample image. Exemplarily, this cosine similarity distance satisfies the formula: A represents fake_id_features, B represents source_id_feature, ||A|| represents the norm of fake_id_feature, and ||B|| represents the norm of source_id_feature.

[0116] 504. Update the model parameters of the initial generation model based on the first loss value, the second loss value, and the third loss value to obtain the target generation model.

[0117] In this example, after performing the above steps 501 to 503 to calculate the first loss value (such as Loss1 mentioned above), the second loss value (such as Loss2 mentioned above), and the third loss value (such as ID_Loss mentioned above), respectively, the sum of the first loss value, the second loss value, and the third loss value can be used to update the model parameters of the initial generation model to train the target generation model. Exemplarily, iterative training optimization can be performed on the sum of the first loss value, the second loss value, and the third loss value based on the gradient descent algorithm until the training result meets the conditions for terminating the model training. It should be noted that the described gradient descent algorithm can include algorithms such as stochastic gradient descent, stochastic gradient descent with momentum, adaptive gradient algorithm, and adaptive moment estimation algorithm, which are not specifically limited in the embodiments of this application. Additionally, the conditions for terminating the model training can include setting the number of training iterations to meet a preset value, or the sum of the loss values being less than a preset value, etc., which are not specifically limited in this application.

[0118] Optionally, the model parameters of the initial generation model can also be updated using the sum of the first loss value after weighted summation, the second loss value after weighted summation, and the third loss value to obtain the target generation model.

[0119] In some other alternative examples, the model parameters of the aforementioned Figure 2 preset discriminant model can also be updated so that the updated discriminant model can more accurately determine the discriminant result of the image, thereby improving the calculation accuracy of the loss value and enhancing the learning ability of the target generation model. As a schematic description, how to update the model parameters of the preset discriminant model can be understood with reference to the following method, that is:

[0120] Based on a preset discrimination model, discriminant processing is respectively performed on the second partial facial feature map, the target sample image, and the label sample image to obtain the discrimination result of the second partial facial feature map, the discrimination result of the target sample image, and the discrimination result of the label sample image. In this way, based on the discrimination result of the first partial facial feature map and the discrimination result of the second partial facial feature map, the loss value of the facial feature discrimination result is calculated. For example, the loss value of the facial feature discrimination result satisfies: eye_mouth_D_loss = -log(eye_mouth_D(GT)) - log(1 – eye_mouth_D(fake)). Wherein, eye_mouth_D(GT) represents the discrimination result of the second partial facial feature map, and eye_mouth_D(fake) represents the discrimination result of the first partial facial feature map.

[0121] Similarly, it is also necessary to calculate the loss value of the image discrimination result according to the discrimination result of the target sample image and the discrimination result of the label sample image. For example, the loss value of the image discrimination result satisfies: D_loss = -logD(GT) - log(1 – D(fake)). Wherein, D(GT) represents the discrimination result of the label sample image, and D(fake) represents the discrimination result of the target sample image.

[0122] After obtaining the loss value of the facial feature discrimination result and the loss value of the image discrimination result, based on the loss value of the facial feature discrimination result and the loss value of the image discrimination result, the model parameters of the preset discrimination model are updated. For example, weighted summation processing can be performed on the loss value of the facial feature discrimination result and the loss value of the image discrimination result, so as to update the model parameters of the preset discrimination model by using the weighted summation result loss value to obtain an updated preset discrimination model. In this way, through the updated preset discrimination model, result judgment processing can be respectively performed on the target sample image and the first partial facial feature map in the next training iteration process.

[0123] It should be noted that for the preset discrimination model used for judging the partial facial feature map and the preset discrimination model used for judging the sample image, they can be discriminators with the same structure or discriminators with different structures, which are not specifically limited in this application.

[0124] The above Figure 3 Mainly from the perspective of method embodiments, the method for training the model provided by the embodiments of this application is described. After training the target generation model based on Figure 3 the described method, in the method of performing image processing, the target generation model can be used to generate the corresponding target facial image. Exemplarily, based on the target generation model shown in the foregoing Figure 3 and Figure 6 the flowchart of the method for image processing provided by the embodiments of this application shown. As Figure 6As shown, the image processing method provided by the present application comprises at least the following steps:

[0125] 601. Acquire a first facial image and a second facial image, where the first facial image and the second facial image are different.

[0126] In this example, the first facial image can be understood as an image that needs to be replaced with a face. Figure 3 The first facial image is understood by the source sample image described in step 301. The difference between the first facial image and the source sample image is that the source sample image is the facial image used in the model training phase, and the first facial image is the facial image used in the model use phase.

[0127] Exemplarily, the method for obtaining the first facial image is not limited in this application. For example, it can be obtained from facial images stored in a database. Alternatively, the object can be photographed, or the facial image of the object can be captured from a video. Alternatively, the original image of the object can be first obtained, and facial detection can be performed on the original image using face detection or other methods to obtain the facial area in the original image; further, facial alignment can be performed in the facial area of ​​the original image to obtain multiple facial key points, such as eye key points, mouth corner key points, etc.; finally, the original image is captured based on the multiple facial key points to obtain the first facial image. It should be noted that the difference between the original image and the first facial image mentioned here is that the first facial image can be understood as the facial image in the facial area of ​​the original image.

[0128] It should be noted that in practical applications, other methods of obtaining the first facial image may also be included, which are not specifically limited in this application.

[0129] The second facial image mentioned above can be understood as a template image for face replacement. Figure 3 The second facial image mentioned here is different from the template sample image in that the template sample image is a template image used in the model training phase, and the second facial image is a template image used in the model use phase.

[0130] It should be noted that the method for obtaining the second facial image can be understood by referring to the aforementioned method for obtaining the first facial image, and the details will not be repeated here.

[0131] 602. Replace the facial region of the first facial image with the facial region of the second facial image based on the target generation model to obtain a target facial image. The target generation model is a machine learning model obtained by training an initial generation model with source sample images, target sample images, label sample images, the first local facial features map of the target sample image, and the second local facial features map of the label sample image. The target sample image is an image obtained by replacing the facial region in the source sample image with the facial region in the template sample image using the initial generation model.

[0132] In this example, for the described target generation model, the model training process can specifically refer to the content shown in the foregoing Figures 3 to 5 and will not be elaborated here.

[0133] After obtaining the first facial image and the second facial image in step 601, use the first facial image and the second facial image as the input of the target generation model, so that the target generation model can replace the facial region of the first facial image with the facial region of the second facial image, thereby generating a target facial image.

[0134] Exemplarily, in order to make the face in the generated target facial image closer to the facial details such as facial features in the first facial image, during the process of generating the target facial image using the target generation model, the identity features of the first facial image can also be used for constraint. That is, before replacing the facial region of the first facial image with the facial region of the second facial image using the target generation model, first extract the identity features of the first facial image through a model such as a face recognition network. It should be noted that the face recognition network mentioned here can be understood with reference to the content described in step 302 of the foregoing Figure 3 and will not be elaborated here.

[0135] In this way, during the process of generating the target facial image using the target generation model, specifically, first encode the first facial image and the second facial image based on the encoding module in the target generation model to obtain encoded feature vectors. For example, the first facial image and the second facial image can be stitched or overlapped, and then the stitched or overlapped image is successively encoded through multiple convolutional layers in the encoding module to obtain the encoded feature vector.

[0136] After obtaining the encoded feature vector, the identity feature of the first facial image is taken into consideration to perform feature fusion on the encoded feature vector and the identity feature of the first facial image, thereby obtaining a fused feature vector. As a schematic description of how to obtain the fused feature vector, one can calculate the feature mean of the encoded feature vector, the feature standard deviation of the encoded feature vector, the feature mean of the identity feature of the first facial image, and the feature standard deviation of the identity feature; further, based on the encoded feature vector, the feature mean of the encoded feature vector, the feature standard deviation of the encoded feature vector, the feature mean of the identity feature of the first facial image, and the feature standard deviation of the identity feature, the fused feature vector is determined. It should be noted that for how to calculate the fused feature vector here, one can also refer to the process of calculating the second feature vector in step 302 mentioned above for understanding, and details are not elaborated here. Figure 3 for understanding, details are not elaborated here.

[0137] In this way, after obtaining the fused feature vector, the fused feature vector is used as the input of the decoding module in the target generation model to perform decoding processing on the encoded feature vector through the decoding module in the target generation model, for example, performing deconvolution operation to generate a target facial image.

[0138] Optionally, after generating the target facial image, the target facial image can also be displayed.

[0139] It should be noted that for the encoding module and the decoding module in the target generation model mentioned above, one can also refer to the schematic diagrams of the encoding module and the decoding module shown above for understanding, and details are not specifically elaborated here. Figure 4 for understanding, details are not specifically elaborated here.

[0140] In the embodiments of the present application, during the process of training the target generation model, not only the source sample image, the label sample image, and the target sample image obtained based on the source sample image and the template sample image are considered, but also the first local facial features map of the target sample image and the second local facial features map of the label sample image are taken into consideration, so that the generated target generation model can not only focus on and optimize the overall facial image, but also focus on the local details of the facial features in the image, thereby completing the detail optimization processing of the local details of the facial features. In this way, during the subsequent process of using the target generation model to process the first facial image and the second facial image, the generation quality of the generated target facial image can be greatly improved, and the facial replacement effect can be enhanced.

[0141] The above mainly introduces the solution provided by the embodiments of the present application from the perspective of methods. It can be understood that in order to implement the above functions, the corresponding hardware structures and / or software modules for executing each function are included. Those skilled in the art should easily realize that, in combination with the modules and algorithm steps of each example described in the embodiments disclosed in the present application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving the hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0142] The embodiments of the present application can divide the device into functional modules according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present application is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0143] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of the overall module or unit that includes the function of the module or unit.

[0144] The image processing device in the embodiments of the present application will be described in detail below. Figure 7 It is a schematic diagram of an embodiment of the image processing device provided in the embodiments of the present application. As Figure 7 shown, the image processing device may include an acquisition unit 701 and a processing unit 702.

[0145] Among them, the acquisition unit 701 is used to acquire a first facial image and a second facial image, and the first facial image and the second facial image are different. Specifically, reference can be made to the content described in step 601 in the foregoing Figure 6 for understanding, and details are not described here.

[0146] A processing unit 702 is configured to replace the facial region of the first facial image with the facial region of the second facial image based on a target generation model to obtain a target facial image. The target generation model is a machine learning model obtained by training an initial generation model using source sample images, target sample images, label sample images, first partial facial feature maps of the target sample images, and second partial facial feature maps of the label sample images as training data. The target sample image is an image obtained by replacing the facial region in the source sample image with the facial region in the template sample image using the initial generation model. For specific details, reference may be made to the content described in step 602 above, which will not be elaborated here. Figure 6 for understanding in accordance with the content described in step 602 above, which will not be elaborated here.

[0147] In some alternative embodiments, the processing unit 702 is further configured to extract the identity feature of the first facial image before replacing the facial region of the first facial image with the facial region of the second facial image based on the target generation model to obtain a target facial image. The processing unit 702 is configured to: perform encoding processing on the first facial image and the second facial image based on the target generation model to obtain encoded feature vectors; perform feature fusion on the encoded feature vectors and the identity feature of the first facial image to obtain fused feature vectors; and perform decoding processing on the fused feature vectors based on the target generation model to generate a target facial image.

[0148] In some other alternative embodiments, the processing unit 702 is configured to: calculate the feature mean of the encoded feature vectors and the feature standard deviation of the encoded feature vectors; calculate the feature mean of the identity feature of the first facial image and the feature standard deviation of the identity feature; and determine the fused feature vectors based on the encoded feature vectors, the feature mean of the encoded feature vectors, the feature standard deviation of the encoded feature vectors, the feature mean of the identity feature of the first facial image, and the feature standard deviation of the identity feature.

[0149] The above mainly describes the image processing apparatus in the embodiments of the present application from the perspective of functional modules. Next, the model training apparatus in the embodiments of the present application will be described from the perspective of functional models. Figure 8 FIG. shows a schematic diagram of the functional modules of the model training apparatus provided in the embodiments of the present application. As Figure 8 shown, the model training apparatus includes an acquisition module 801 and a processing module 802.

[0150] Among them, the acquisition module 801 is configured to acquire source sample images, template sample images, and label sample images, where the source sample images are different from the template sample images. For specific details, reference may be made to the content described in step 301 above, which will not be elaborated here. Figure 3 for understanding in accordance with the content described in step 301 above, which will not be elaborated here.

[0151] The processing module 802 is configured to replace the facial region in the source sample image with the facial region in the template sample image based on the initial generation model to generate a target sample image. Specifically, reference may be made to the content described in step 302 above Figure 3 for understanding, and details are not described here.

[0152] The processing module 802 is configured to extract the first local facial feature map of the target sample image and the second local facial feature map of the label sample image. Specifically, reference may be made to the content described in step 303 above Figure 3 for understanding, and details are not described here.

[0153] The processing module 802 is configured to update the model parameters of the initial generation model based on the first local facial feature map, the second local facial feature map, the target sample image, the label sample image, and the source sample image to obtain a target generation model. The target generation model is used to replace the facial region of the first facial image with the facial region of the second facial image to obtain a target facial image, where the first facial image and the second facial image are different. Specifically, reference may be made to the content described in step 304 above Figure 3 for understanding, and details are not described here.

[0154] In some alternative embodiments, the processing module 802 is further configured to extract the identity feature of the source sample image before replacing the facial region in the source sample image with the facial region in the template sample image based on the initial generation model to generate a target sample image. The processing unit is configured to: perform encoding processing on the facial region in the source sample image and the facial region in the template sample image based on the encoding module in the initial generation model to obtain a first feature vector; perform feature fusion on the first feature vector and the identity feature to obtain a second feature vector; perform decoding processing on the second feature vector based on the decoding module in the initial generation model to generate a target sample image.

[0155] In some other alternative embodiments, the processing module 802 is configured to: calculate the image difference between the first local facial feature map and the second local facial feature map to obtain a first loss value; calculate the image difference between the target sample image and the label sample image to obtain a second loss value; calculate the image difference between the target sample image and the source sample image to obtain a third loss value; update the model parameters of the initial generation model based on the first loss value, the second loss value, and the third loss value to obtain a target generation model.

[0156] In some other alternative embodiments, the processing module 802 is configured to: calculate the pixel difference between the first partial facial feature map and the second partial facial feature map based on the pixel information of the first partial facial feature map and the pixel information of the second partial facial feature map, so as to obtain a first pixel difference value; extract multiple facial features of the first partial facial feature map, and extract multiple facial features of the second partial facial feature map, and calculate the facial feature difference value between the multiple facial features of the first partial facial feature map and the multiple facial features of the second partial facial feature map; perform a discrimination process on the first partial facial feature map based on a preset discrimination model, so as to obtain a discrimination result of the first partial facial feature map; calculate a result loss value of the first partial facial feature map based on the discrimination result of the first partial facial feature map; calculate the sum of the first pixel difference value, the facial feature difference value, and the result loss value of the first partial facial feature map, so as to obtain a first loss value.

[0157] In some other alternative embodiments, the processing module 802 is configured to: calculate the absolute feature difference between each facial feature in the first partial facial feature map and the corresponding facial feature in the second partial facial feature map, to obtain multiple absolute feature differences; calculate the sum of the multiple absolute feature differences, to obtain the facial feature difference value between the multiple facial features of the first partial facial feature map and the multiple facial features of the second partial facial feature map.

[0158] In some other alternative embodiments, the processing module 802 is configured to: perform feature extraction processing on the first partial facial feature map based on a preset feature extraction model, to obtain multiple facial features of the first partial facial feature map; perform feature extraction processing on the second partial facial feature map based on a preset feature extraction model, to obtain multiple facial features of the second partial facial feature map.

[0159] In some other alternative embodiments, the processing module 802 is configured to: calculate the pixel difference between the target sample image and the label sample image based on the pixel information of the target sample image and the pixel information of the label sample image, so as to obtain a second pixel difference value; extract multiple image features of the target sample image, and extract multiple image features of the label sample image, and calculate the feature difference value between the multiple image features of the target sample image and the multiple image features of the label sample image; perform a discrimination process on the target sample image based on a preset discrimination model, so as to obtain a discrimination result of the target sample image; calculate a result loss value of the target sample image based on the discrimination result of the target sample image; calculate the sum of the second pixel difference value, the feature difference value, and the result loss value of the target sample image, so as to obtain a second loss value.

[0160] In some other alternative embodiments, the processing module 802 is configured to: calculate the absolute feature differences between multiple image features of the target sample image and the corresponding image features in the labeled sample image to obtain multiple absolute image feature differences; calculate the sum of the multiple absolute image feature differences, and calculate the feature difference value between the multiple image features of the target sample image and the multiple image features of the labeled sample image.

[0161] In some other alternative embodiments, the processing module 802 is configured to: extract the identity features of the target sample image and the identity features of the source sample image; calculate the similarity distance between the identity features of the target sample image and the identity features of the source sample image to obtain the similarity between the identity features of the target sample image and the identity features of the source sample image; and obtain a third loss value based on the similarity.

[0162] In some other alternative embodiments, the processing module 802 is further configured to: perform discrimination processing on the second partial facial feature map, the target sample image, and the labeled sample image respectively based on a preset discrimination model to obtain the discrimination result of the second partial facial feature map, the discrimination result of the target sample image, and the discrimination result of the labeled sample image; calculate the facial feature discrimination result loss value based on the discrimination result of the first partial facial feature map and the discrimination result of the second partial facial feature map, and calculate the image discrimination result loss value based on the discrimination result of the target sample image and the discrimination result of the labeled sample image; and update the model parameters of the preset discrimination model based on the facial feature discrimination result loss value and the image discrimination result loss value.

[0163] In some other alternative embodiments, the processing module 802 is configured to: perform facial feature region segmentation processing on the target sample image based on a preset segmentation model to obtain the facial feature key points in the target sample image, and obtain a first partial facial feature map based on the facial feature key points in the target sample image; perform facial feature region segmentation processing on the labeled sample image based on the preset segmentation model to obtain the facial feature key points in the source sample image, and obtain a second partial facial feature map based on the facial feature key points in the labeled sample image.

[0164] The above describes the image processing apparatus and the model training apparatus in the embodiments of the present application from the perspective of modular functional entities. Next, the image processing device in the embodiments of the present application will be described from the perspective of hardware processing. Figure 9 It is a schematic hardware structure diagram of the image processing device provided in the embodiments of the present application. This image processing device may vary greatly due to configuration or performance differences. For example, it may include but is not limited to Figure 7 the image processing apparatus shown in Figure 8 the model training apparatus shown in

[0165] Such as Figure 9As shown, the image processing device 300 can vary significantly depending on its configuration or performance, and may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and a memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 342 or data 344. Among them, the memory 332 and the storage media 330 can be transient storage or persistent storage. The programs stored in the storage media 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the image processing device. Further, the central processing unit 322 can be configured to communicate with the storage media 330 and execute a series of instruction operations in the storage media 330 on the image processing device 300. Exemplarily, the central processing unit 322 is used to execute the computer-executable instructions stored in the storage media 330, so as to implement the model training method or the image processing method provided in the above embodiments of the present application.

[0166] The image processing device 300 may further include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0167] Exemplarily, Figure 9 the central processing unit 322 in can cause the image processing device to execute the method in the corresponding method embodiment by calling the computer-executable instructions stored in the memory 332. Figures 3 to 6

[0168] Specifically, Figure 8 the functions / implementation processes of the processing module 802 in, and Figure 7 the processing unit 702 in can be implemented by the central processing unit 322 in calling the computer-executable instructions stored in the memory 332. Figure 9 Figure 8 the functions / implementation processes of the acquisition module 801 in and Figure 7 the acquisition unit 701 in can be implemented by the input / output interface 358 in. Figure 9

[0169] The steps performed by the image processing device in the above embodiments can be based on the structure of the image processing device shown in this Figure 9

[0170] ​​​​In an embodiment of the present application, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the methods described in the foregoing embodiments are implemented.

[0171] In an embodiment of the present application, a computer program product is further provided, including a computer program. When the computer program is executed by a processor, the steps of the methods described in the foregoing embodiments are implemented.

[0172] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0173] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of devices or units can be in electrical, mechanical, or other forms.

[0174] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] In addition, each functional unit in each embodiment of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0176] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0177] The computer program product includes one or more computer instructions. When the computer execution instructions are loaded and executed on a computer, they generate all or part of the processes or functions according to the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center to another website, a computer, a server, or a data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, SSD), etc.

[0178] The above embodiments are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of various embodiments of this application.

Claims

1. A method for image processing, characterized in that, Including: Obtain a first facial image and a second facial image, where the first facial image and the second facial image are different; Based on a target generation model, replace the facial region of the first facial image with the facial region of the second facial image to obtain a target facial image. The target generation model is a machine learning model obtained by training an initial generation model with source sample images, target sample images, label sample images, a first local facial feature map of the target sample image, and a second local facial feature map of the label sample image. The target sample image is an image obtained by the initial generation model replacing the facial region in the source sample image with the facial region in a template sample image.

2. The method according to claim 1, wherein Before replacing the facial region of the first facial image with the facial region of the second facial image based on the target generation model to obtain a target facial image, the method further includes: Extract the identity feature of the first facial image; Replacing the facial region of the first facial image with the facial region of the second facial image based on the target generation model to obtain a target facial image includes: Perform encoding processing on the first facial image and the second facial image based on the target generation model to obtain encoded feature vectors; Perform feature fusion on the encoded feature vectors and the identity feature of the first facial image to obtain fused feature vectors; Perform decoding processing on the fused feature vectors based on the target generation model to generate a target facial image.

3. The method according to claim 2, wherein Performing feature fusion on the encoded feature vectors and the identity feature of the first facial image to obtain fused feature vectors includes: Calculate the feature mean of the encoded feature vectors and the feature standard deviation of the encoded feature vectors; Calculate the feature mean of the identity feature of the first facial image and the feature standard deviation of the identity feature; Determine the fused feature vectors based on the encoded feature vectors, the feature mean of the encoded feature vectors, the feature standard deviation of the encoded feature vectors, the feature mean of the identity feature of the first facial image, and the feature standard deviation of the identity feature.

4. A method for model training, characterized in that, Including: Obtain a source sample image, a template sample image, and a label sample image, where the source sample image is different from the template sample image; Based on the initial generation model, replace the facial region in the source sample image with the facial region in the template sample image to generate a target sample image; Extract a first local facial feature map of the target sample image and a second local facial feature map of the label sample image; Based on the first local facial feature map, the second local facial feature map, the target sample image, the label sample image, and the source sample image, update the model parameters of the initial generation model to obtain a target generation model. The target generation model is used to replace the facial region of a first facial image with the facial region of a second facial image to obtain a target facial image, where the first facial image is different from the second facial image.

5. The method according to claim 4, characterized in that, Before replacing the facial region in the source sample image with the facial region in the template sample image based on the initial generation model to generate a target sample image, the method further includes: Extracting the identity features of the source sample image; Replacing the facial region in the source sample image with the facial region in the template sample image based on the initial generation model to generate a target sample image, including: Performing encoding processing on the facial region in the source sample image and the facial region in the template sample image based on the encoding module in the initial generation model to obtain a first feature vector; Performing feature fusion on the first feature vector and the identity features to obtain a second feature vector; Performing decoding processing on the second feature vector based on the decoding module in the initial generation model to generate a target sample image.

6. The method according to any one of claims 4 to 5, characterized in that, Updating the model parameters of the initial generation model based on the first local facial feature map, the second local facial feature map, the target sample image, the labeled sample image, and the source sample image to obtain a target generation model, including: Calculating the image difference between the first local facial feature map and the second local facial feature map to obtain a first loss value; Calculating the image difference between the target sample image and the labeled sample image to obtain a second loss value; Calculating the image difference between the target sample image and the source sample image to obtain a third loss value; Updating the model parameters of the initial generation model based on the first loss value, the second loss value, and the third loss value to obtain a target generation model.

7. The method according to claim 6, wherein Calculating the image difference between the first local facial feature map and the second local facial feature map to obtain a first loss value, including: Calculating the pixel difference between the first local facial feature map and the second local facial feature map based on the pixel information of the first local facial feature map and the pixel information of the second local facial feature map to obtain a first pixel difference value; Extracting multiple facial feature points of the first local facial feature map and multiple facial feature points of the second local facial feature map, and calculating the facial feature difference value between the multiple facial feature points of the first local facial feature map and the multiple facial feature points of the second local facial feature map; Performing discrimination processing on the first local facial feature map based on a preset discrimination model to obtain the discrimination result of the first local facial feature map; Calculating the result loss value of the first local facial feature map based on the discrimination result of the first local facial feature map; Calculating the sum of the first pixel difference value, the facial feature difference value, and the result loss value of the first local facial feature map to obtain a first loss value.

8. The method according to claim 7, characterized in that, Calculating the facial feature difference value between the multiple facial feature points of the first local facial feature map and the multiple facial feature points of the second local facial feature map, including: Calculating the absolute feature difference between each facial feature point in the first local facial feature map and the corresponding facial feature point in the second local facial feature map to obtain multiple absolute feature difference values; Calculating the sum of the multiple absolute feature difference values to obtain the facial feature difference value between the multiple facial feature points of the first local facial feature map and the multiple facial feature points of the second local facial feature map.

9. The method according to claim 6, characterized in that, Calculating the image difference between the target sample image and the labeled sample image to obtain a second loss value, including: Calculating the pixel difference between the target sample image and the labeled sample image based on the pixel information of the target sample image and the pixel information of the labeled sample image to obtain a second pixel difference value; Extracting multiple image features of the target sample image and multiple image features of the labeled sample image, and calculating the feature difference value between the multiple image features of the target sample image and the multiple image features of the labeled sample image; Performing a discrimination process on the target sample image based on a preset discrimination model to obtain the discrimination result of the target sample image; Calculating the result loss value of the target sample image based on the discrimination result of the target sample image; Calculating the sum of the second pixel difference value, the feature difference value, and the result loss value of the target sample image to obtain a second loss value.

10. The method according to claim 6, wherein Calculating the image difference between the target sample image and the source sample image to obtain a third loss value, including: Extracting the identity feature of the target sample image and the identity feature of the source sample image; Calculating the similarity distance between the identity feature of the target sample image and the identity feature of the source sample image to obtain the similarity between the identity feature of the target sample image and the identity feature of the source sample image; Obtaining a third loss value based on the similarity.

11. An image processing apparatus, characterized in that, Including: An acquisition unit for acquiring a first facial image and a second facial image, wherein the first facial image and the second facial image are different; A processing unit for replacing the facial region of the first facial image with the facial region of the second facial image based on a target generation model to obtain a target facial image, where the target generation model is a machine learning model obtained by training an initial generation model with the source sample image, the target sample image, the labeled sample image, the first local facial features map of the target sample image, and the second local facial features map of the labeled sample image, and the target sample image is an image obtained by replacing the facial region in the source sample image with the facial region in the template sample image by the initial generation model.

12. A model training device, characterized in that, Including: An acquisition module for acquiring a source sample image, a template sample image, and a labeled sample image, wherein the source sample image is different from the template sample image; A processing module for replacing the facial region in the source sample image with the facial region in the template sample image based on the initial generation model to generate a target sample image; The processing module for extracting the first local facial features map of the target sample image and the second local facial features map of the labeled sample image; The processing module is configured to update model parameters of the initial generation model based on the first partial facial feature map, the second partial facial feature map, the target sample image, the labeled sample image, and the source sample image, so as to obtain a target generation model, where the target generation model is used to replace the facial region of a first facial image with the facial region of a second facial image to obtain a target facial image, and the first facial image is different from the second facial image.

13. A voice recognition device, characterized in that, It includes: an input / output interface, a processor, and a memory, where program instructions are stored in the memory; The processor is configured to execute the program instructions stored in the memory to perform the image processing method according to any one of claims 1 to 3; or execute the model training method according to any one of claims 4 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when running on a computer device, cause the computer device to execute the image processing method according to any one of claims 1 to 3; or execute the model training method according to any one of claims 4 to 10.

15. A computer program product, characterized in that, The computer program product includes instructions that, when running on a computer device, cause the computer device to execute the image processing method according to any one of claims 1 to 3; or execute the model training method according to any one of claims 4 to 10.

Images