Face replacement method and related device

Abstract

The invention discloses a face replacement method and a related device. Aligning a first source face in the first source face image with a first target face in the first target face image to obtain a second source face image, wherein a face image and a face posture of a second source face in the second source face image are a face image of the first source face and a face posture of the first target face respectively; according to the target center point and the target face shape key point of the first target face and the source center point source and the face shape key point of the second source face, the face shape difference quantity of the first target face and the second source face is calculated; and according to an adjustment direction from the target facial form key point to the target central point, adjusting target pixel points in a target adjustment area determined by the target facial form key point and the facial form difference quantity based on backward mapping pixels to obtain a second target facial image, the target face shape of the second target face in the second target face image is the source face shape of the second source face; and replacing the second target face with the second source face to obtain a replaced face image.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and particularly to a face replacement method and related device. Background Art

[0002] Face replacement technology is an image processing technology. Replacing a target face with a source face means replacing the face image of the source face onto the target face while maintaining the face pose of the target face, achieving the face replacement effect.

[0003] In related technologies, a face replacement method refers to, after identifying the source face key points of the source face and the target face key points of the target face, calculating a transformation matrix through the source face key points and the target face key points, so that the source face is transformed into the face pose of the target face to replace the target face.

[0004] However, when the source face cannot completely cover the target face, obvious defects are likely to appear at the face edge of the replaced face obtained by using the above face replacement method, resulting in a poor face replacement effect. Summary of the Invention

[0005] To solve the above technical problems, the present application provides a face replacement method and related device, so that there are no obvious defects at the face edge of the replaced face in the replaced face image, thereby improving the face replacement effect.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] On the one hand, the embodiments of the present application provide a face replacement method, and the method includes:

[0008] Performing face alignment on a first source face in a first source face image according to a first target face in a first target face image to obtain a second source face image; the face image of the second source face in the second source face image is the face image of the first source face, and the face pose of the second source face is the face pose of the first target face;

[0009] Calculating a difference between the first target face and the second source face to obtain a face shape difference amount between the first target face and the second source face according to the target center point of the first target face, the target face shape key points, the source center point of the second source face, and the source face shape key points;

[0010] Performing pixel adjustment based on backward mapping on target pixels in a target adjustment area of the target face shape key points according to the adjustment direction from the target face shape key points to the target center point to obtain a second target face image; the target adjustment area is determined according to the target face shape key points and the face shape difference amount, and the target face shape of the second target face in the second target face image is the source face shape of the second source face;

[0011] Perform face replacement on the second target face according to the second source face to obtain a replaced face image.

[0012] On the other hand, an embodiment of the present application provides a face replacement device, which includes: an alignment unit, a calculation unit, an adjustment unit, and a replacement unit;

[0013] The alignment unit is configured to perform face alignment on the first source face in the first source face image according to the first target face in the first target face image to obtain a second source face image; the face image of the second source face in the second source face image is the face image of the first source face, and the face pose of the second source face is the face pose of the first target face;

[0014] The calculation unit is configured to perform difference calculation according to the target center point of the first target face, the target face shape key points, the source center point of the second source face, and the source face shape key points to obtain the face shape difference amount between the first target face and the second source face;

[0015] The adjustment unit is configured to perform pixel adjustment based on backward mapping on the target pixel points in the target adjustment area of the target face shape key points according to the adjustment direction from the target face shape key points to the target center point to obtain a second target face image; the target adjustment area is determined according to the target face shape key points and the face shape difference amount, and the target face shape of the second target face in the second target face image is the source face shape of the second source face;

[0016] The replacement unit is configured to perform face replacement on the second target face according to the second source face to obtain a replaced face image.

[0017] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory:

[0018] The memory is used to store a computer program and transmit the computer program to the processor;

[0019] The processor is configured to execute the method described in any of the foregoing aspects according to the instructions in the computer program.

[0020] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which is used to store a computer program. When the computer program runs on a computer device, the computer device is enabled to execute the method described in any of the foregoing aspects.

[0021] On the other hand, an embodiment of the present application provides a computer program product, including a computer program, which, when running on a computer device, causes the computer device to execute the method described in any of the foregoing aspects.

[0022] As can be seen from the above technical solutions, first, the first source face in the first source face image is aligned with the first target face in the first target face image to obtain a second source face image. The face image of the second source face in the second source face image is the face image of the first source face, and the face pose of the second source face is the face pose of the first target face. This method converts the face pose of the first source face to the face pose of the first target face in the first target face image on the basis of retaining the face image of the first source face in the first source face image, so as to obtain the second source face in the second source face image, making the face image of the second source face consistent with the face image of the first source face and the face pose of the second source face consistent with the face pose of the first target face.

[0023] Then, the face shape difference amount between the first target face and the second source face is calculated through the target center point of the first target face, the target face shape key points, the source center point of the second source face, and the source face shape key points. According to the adjustment direction from the target face shape key points to the target center point, pixel adjustment based on backward mapping is performed on the target pixel points in the target adjustment area determined by the target face shape key points and the face shape difference amount to obtain a second target face image. The target face shape of the second target face in the second target face image is the source face shape of the second source face. This method takes into account the face shape difference between the target face shape of the first target face in the first target face image and the source face shape of the second source face in the second source face image, and adjusts the target face shape of the first target face to the source face shape of the second source face, so as to obtain the second target face in the second target face image, making the target face shape of the second target face consistent with the source face shape of the second source face.

[0024] Finally, the second target face is replaced with the second source face to obtain a replaced face image. This method replaces the second target face with the second source face in the second source face image on the basis that the target face shape of the second target face in the second target face image is consistent with the source face shape of the second source face, making the face edge of the replaced face in the replaced face image have no obvious defects, thereby improving the face replacement effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0026] Figure 1 It is a schematic diagram of a source face image, a target face image, and a replaced face image in the related art;

[0027] Figure 2 It is a schematic system diagram of a face replacement method provided by an embodiment of the present application;

[0028] Figure 3 It is a flowchart of a face replacement method provided by an embodiment of the present application;

[0029] Figure 4 It is a schematic diagram of obtaining a second source face image by aligning a first source face in a first source face image with a first target face in a first target face image provided by an embodiment of the present application;

[0030] Figure 5 It is a schematic diagram of a replaced face image provided by an embodiment of the present application;

[0031] Figure 6 It is a schematic diagram of performing backward mapping on target pixels in a target adjustment area to obtain mapped pixels according to the adjustment direction from target face shape key points to the target center point, through the target face shape key points, the target center point, and the target radius of the target adjustment area provided by an embodiment of the present application;

[0032] Figure 7 It is a schematic diagram of performing interpolation processing in the X-axis direction and the Y-axis direction on multiple pixel values corresponding to multiple nearest neighbor pixels of a mapped pixel to obtain a resampled pixel value corresponding to a target pixel provided by an embodiment of the present application;

[0033] Figure 8 It is a schematic diagram of multiple face key points provided by an embodiment of the present application;

[0034] Figure 9 It is a schematic diagram of horizontalizing a source image to be processed according to multiple first face key points to obtain a horizontalized source image provided by an embodiment of the present application;

[0035] Figure 10 It is a schematic diagram of cropping a first source face in a horizontalized source image according to multiple first face key points to obtain a first source face image provided by an embodiment of the present application;

[0036] Figure 11 It is a schematic diagram of face replacement in a face swapping program provided by an embodiment of the present application;

[0037] Figure 12 It is a structural diagram of a face replacement device provided by an embodiment of the present application;

[0038] Figure 13The structural diagram of a server provided by an embodiment of the present application;

[0039] Figure 14 The structural diagram of a terminal provided by an embodiment of the present application. Detailed implementation manners

[0040] The embodiments of the present application will be described below with reference to the accompanying drawings.

[0041] At present, the face replacement method refers to calculating a transformation matrix through the source face key points and the target face key points after identifying the source face key points of the source face and the target face key points of the target face, so that the source face is transformed into the face pose of the target face to replace the target face. For example, if the source face is face A and the target face is face B, replacing face B with face A means that after identifying the face key points of face A and the face key points of face B, calculating the transformation matrix makes face A transform into the face pose of face B to replace face B.

[0042] However, through research, it is found that when the source face cannot completely cover the target face, obvious defects are likely to appear on the face edge of the replaced face obtained by using the above face replacement method, resulting in a poor face replacement effect. Refer to Figure 1 , Figure 1 which is a schematic diagram of a source face image, a target face image and a replaced face image in the related art; where Figure 1 in (a) represents the source face image, and the source face in the source face image is face A, Figure 1 in (b) represents the target face image, and the target face in the target face image is face B, Figure 1 in (c) represents the replaced face image. Face A cannot completely cover face B. Using the above face replacement method to implement the replacement of face B with face A, obvious defects are likely to appear on the face edge of the replaced face in the obtained replaced face image, that is, there are large defects on the left chin 101 and the ear edge 102 of the replaced face, resulting in a poor face replacement effect of replacing face B with face A.

[0043] An embodiment of the present application provides a face replacement method. Based on retaining the face image of the first source face in the first source face image, the face pose of the first source face is converted into the face pose of the first target face in the first target face image to obtain the second source face in the second source face image, so that the face image of the second source face is consistent with the face image of the first source face, and the face pose of the second source face is consistent with the face pose of the first target face. Considering the face shape difference between the target face shape of the first target face in the first target face image and the source face shape of the second source face in the second source face image, the target face shape of the first target face is adjusted to the source face shape of the second source face to obtain the second target face in the second target face image, so that the target face shape of the second target face is consistent with the source face shape of the second source face. Then, the second target face is replaced with the second source face in the second face image, so that there are no obvious defects in the face edge of the replaced face in the replaced face image, thereby improving the face replacement effect.

[0044] Next, the system architecture of the face replacement method will be introduced. Refer to Figure 2 , Figure 2 FIG. is a schematic diagram of a system for a face replacement method provided by an embodiment of the present application. The system includes a computer device 200, and the computer device 200 is used to execute the face replacement method.

[0045] The computer device 200 performs face alignment on the first source face in the first source face image according to the first target face in the first target face image to obtain a second source face image. The face image of the second source face in the second source face image is the face image of the first source face, and the face pose of the second source face is the face pose of the first target face.

[0046] As an example, the first source face in the first source face image is face A in face image A, and the first target face in the first target face image is face B in face image B. The computer device 200 aligns face A in face image A with face B in face image B to obtain a second source face image as face image A'. Then, the second source face in the second source face image is face A' in face image A'. The face image of this face A' is the face image of face A, and the face pose of this face A' is the face pose of face B.

[0047] The computer device 200 calculates the difference between the first target face and the second source face according to the target center point of the first target face, the target face shape key points, the source center point of the second source face, and the source face shape key points to obtain the face shape difference amount between the first target face and the second source face.

[0048] As an example, the face key points of face B are B i , i = 0, 1,..., 67, and the face key points of face A' are A i＇ ; Based on the above example, the target center point of face B is B 30 , and the target facial key points of face B are B j , where j = 0, 1, ……, 16, the target center point of face A' is A3 ＇ 0, and the target facial key points of face A' are A j ＇ ; The computer device 200 passes through B 30 、B j 、A3 ＇ 0 and A j ＇ , and calculates that the facial difference amount between face B and face A' is Diff j .

[0049] The computer device 200 performs pixel adjustment based on backward mapping on the target pixel points within the target adjustment area of the target facial key points according to the adjustment direction from the target facial key points to the target center point, and obtains a second target facial image; the target adjustment area is determined according to the target facial key points and the facial difference amount, and the target facial shape of the second target face in the second target facial image is the source facial shape of the second source face.

[0050] As an example, based on the above example, the computer device 200 follows the adjustment direction from B j to B 30 , and performs pixel adjustment based on backward mapping on the target pixel points within the target adjustment area determined by B j and Diff j , and the obtained second target facial image is face image B', then the second target face in the second target facial image is face B' in face image B', and the target facial shape of this face B' is the source facial shape of face A'.

[0051] The computer device 200 performs facial replacement on the second target face according to the second source face, and obtains the replaced facial image.

[0052] As an example, based on the above example, the computer device 200 replaces face B' with face A', and obtains the replaced facial image.

[0053] That is to say, the face replacement method provided by the embodiments of the present application, while retaining the facial image of the first source face in the first source face image, converts the facial pose of the first source face into the facial pose of the first target face in the first target face image to obtain the second source face in the second source face image, so that the facial image of the second source face is consistent with the facial image of the first source face, and the facial pose of the second source face is consistent with the facial pose of the first target face; considering the facial shape difference between the target facial shape of the first target face in the first target face image and the source facial shape of the second source face in the second source face image, adjusts the target facial shape of the first target face to the source facial shape of the second source face to obtain the second target face in the second target face image, so that the target facial shape of the second target face is consistent with the source facial shape of the second source face; thereby replacing the second target face with the second source face in the second face image, so that there are no obvious defects in the facial edge of the replaced face in the replaced face image, thus improving the face replacement effect.

[0054] It should be noted that the image detection method in the embodiments of the present application involves artificial intelligence. Artificial intelligence is to use a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, and is a theory, method, technology and application system that perceives the environment, acquires knowledge and uses knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a way similar to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, enabling the machines to have the functions of perception, reasoning and decision-making.

[0055] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware-level technologies and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model, also known as the large model and the basic model, can be widely applied to downstream tasks in various directions of artificial intelligence after fine-tuning. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. In the embodiments of the present application, artificial intelligence technology mainly involves computer vision technology, and machine learning / deep learning and other technologies.

[0056] Computer vision is a science that studies how to enable machines to "see". More specifically, it refers to machine vision that uses cameras and computers to replace human eyes for target recognition, tracking, and measurement, and further performs graphic processing to make the computer-processed images more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies and attempts to build artificial intelligence systems that can obtain information from images or multi-dimensional data. The large model technology has brought important changes to the development of computer vision technology. Pre-trained models in the field of vision such as swin-transformer, Vision Transformer, Vision MoE, and MAE can be quickly and widely applied to downstream specific tasks after fine-tuning. In the embodiments of this application, computer vision technology mainly involves technologies such as image processing, image recognition, image semantic understanding, image retrieval, optical character recognition, video processing, video semantic understanding, and video content / behavior recognition.

[0057] Machine learning / deep learning is a multi-disciplinary cross-discipline that involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications cover all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning. Pre-trained models are the latest development results of deep learning and integrate the above technologies.

[0058] It should be noted that in the embodiments of this application, the computer device can be a server or a terminal. The method provided in the embodiments of this application can be executed independently by the terminal or the server, or can be executed in cooperation by the terminal and the server. Among them, when the method provided in the embodiments of this application is executed independently by the terminal or the server, its execution method is similar to Figure 2 the corresponding embodiment, mainly replacing the computer device with the terminal or the server. In addition, when the method provided in the embodiments of this application is executed in cooperation by the terminal and the server, the steps that need to be reflected on the front-end interface can be executed by the terminal, while some steps that require background calculation and do not need to be reflected on the front-end interface can be executed by the server.

[0059] Among them, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart voice interaction device, a vehicle-mounted terminal, a smart TV, an extended reality device, an aircraft, etc., but is not limited thereto. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services, but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods, and this application does not make any restrictions here. For example, the terminal and the server can be connected through a network, and the network can be a wired or wireless network.

[0060] In addition, the embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, intelligent transportation, assisted driving, autonomous driving, digital humans, virtual humans, virtual reality, augmented reality, mixed reality, audio and video, etc.

[0061] In addition, for technologies such as face processing based on face images involved in the embodiments of this application, when applying the embodiments of this application to specific products or technologies, the processes of collecting, using, and processing relevant data should comply with the requirements of national laws and regulations. Before collecting face information, the information processing rules should be informed and the separate consent of the target object to which the face information belongs should be solicited, or the legality of collecting face information should be possessed, and the face information should be strictly processed in accordance with the requirements of laws and regulations and personal information processing rules, and technical measures should be taken to ensure the security of relevant data.

[0062] Next, taking a computer device executing the method provided by the embodiments of this application as an example, the face replacement method provided by the embodiments of this application will be introduced in detail in combination with the accompanying drawings. See Figure 3 , Figure 3 which is a flowchart of a face replacement method provided by the embodiments of this application. The method includes:

[0063] S301: Align the first source face in the first source face image with the first target face in the first target face image to obtain a second source face image; the face image of the second source face in the second source face image is the face image of the first source face, and the face pose of the second source face is the face pose of the first target face.

[0064] In the embodiments of this application, considering that replacing the target face with the source face means replacing the face image of the source face onto the target face while retaining the face pose of the target face, for the source face and the target face, it is necessary to align the source face with the target face to retain the face image of the source face and transform the face pose of the source face into the face pose of the target face to obtain the aligned source face.

[0065] Based on this, for replacing the first target face in the first target face image with the first source face in the first source face image, first, it is necessary to align the first source face in the first source face image with the first target face in the first target face image to obtain a second source face image. The facial image of the second source face in the second source face image is the facial image of the first source face, and the facial pose of the second source face is the facial pose of the first target face.

[0066] That is, for replacing the first target face in the first target face image with the first source face in the first source face image, the replaced face in the replaced face image needs to have the facial image of the first source face in the first source face image and the facial pose of the first target face in the first target face image.

[0067] This S301, while preserving the facial image of the first source face in the first source face image, converts the facial pose of the first source face to the facial pose of the first target face in the first target face image to obtain the second source face in the second source face image, so that the facial image of the second source face is consistent with the facial image of the first source face and the facial pose of the second source face is consistent with the facial pose of the first target face; this S301 provides a second source face that conforms to the facial image of the first source face and the facial pose of the first target face for subsequent face replacement.

[0068] As an example of S301, see Figure 4 , Figure 4 which is a schematic diagram provided by an embodiment of the present application for aligning the first source face in the first source face image with the first target face in the first target face image to obtain a second source face image. Among them, Figure 4 (a) represents the first source face image, and the first source face in the first source face image is face A in face image A. Figure 4 In (b), it represents the first target face image, and the first target face in the first target face image is face B in face image B. The computer device aligns face A in face image A with face B in face image B to obtain Figure 4 In (c), it represents the second source face image, and the second source face in the second source face image is face A' in face image A'. The facial image of face A' is the facial image of face A, and the facial pose of face A' is the facial pose of face B.

[0069] S302: Calculate the difference based on the target center point of the first target face, the target facial key points, the source center point of the second source face, and the source facial key points to obtain the facial shape difference amount between the first target face and the second source face.

[0070] S303: Based on the adjustment direction from the target facial key points to the target center point, perform pixel adjustment based on backward mapping on the target pixels within the target adjustment area of the target facial key points to obtain a second target facial image; the target adjustment area is determined according to the target facial key points and the facial shape difference amount, and the target facial shape of the second target face in the second target facial image is the source facial shape of the second source face.

[0071] In the related art, by identifying the source facial key points of the source face and the target facial key points of the target face, calculating a transformation matrix to make the facial pose of the source face transform into that of the target face, and realizing the alignment of the source face to the target face to replace the target face. When the source face cannot completely cover the target face, obvious defects are likely to appear at the facial edges of the replaced face, resulting in a poor facial replacement effect.

[0072] In the embodiments of the present application, to solve the above problems, considering that the source face cannot completely cover the target face, the main reason is that there is a facial shape difference between the target facial shape of the target face and the source facial shape of the source face; then it is necessary to adjust the target facial shape of the target face to the source facial shape of the aligned source face based on the facial shape difference between the target facial shape of the target face and the source facial shape of the source face to obtain the adjusted target face.

[0073] Based on this, after performing S301 to align the first source face in the first source facial image with the first target face in the first target facial image to obtain a second source facial image, it is necessary to adjust the target facial shape of the first target face in the first target facial image to the source facial shape of the second source face in the second source facial image; for adjusting the target facial shape of the first target face to the source facial shape of the second source face, it is necessary to first calculate the facial shape difference between the target facial shape of the first target face and the source facial shape of the second source face, that is, based on the fact that the target facial key points of the first target face represent the target facial shape and the source facial key points of the second source face represent the source facial shape, through the positional relationship between the target center point of the first target face and the target facial key points of the first target face, and the positional relationship between the source center point of the second source face and the source facial key points of the second source face, the facial shape difference amount between the first target face and the second source face can be calculated.

[0074] Then, according to the facial shape difference amount between the first target face and the second source face, adjust the target facial shape of the first target face to the source facial shape of the second source face, that is, based on the fact that adjusting the facial shape means adjusting the pixel value of a pixel point to that of another pixel point through backward mapping in spatial mapping, according to the adjustment direction from the target facial key points to the target center point, for the target pixels within the target adjustment area determined by the target facial key points and the facial shape difference amount, realize pixel adjustment based on backward mapping to realize facial shape adjustment, and obtain a second target facial image; the target facial shape of the second target face in the second target facial image is the source facial shape of the second source face.

[0075] Among them, the target center point of the first target face refers to the face center point of the first target face, which can be the face center point among multiple face key points of the first target face; the target face shape key point of the first target face refers to the face shape point of the first target face, which can be the jaw line key point among multiple face key points of the first target face; the target center point of the first target face refers to the face center point of the first target face, which can be the face center point among multiple face key points of the first target face; the source face shape key point of the first source face refers to the face shape point of the first source face, which can be the jaw line key point among multiple face key points of the first source face.

[0076] Among them, the pixel adjustment based on backward mapping means that according to the adjustment direction from the target face shape key point to the target center point, for the target pixel points in the target adjustment area determined by the target face shape key point and the face shape difference amount, calculate the pixel value of the pixel point mapped to the target pixel point to adjust the pixel value of the target pixel point.

[0077] S302 - S303 takes into account the face shape difference between the target face shape of the first target face in the first target face image and the source face shape of the second source face in the second source face image, and adjusts the target face shape of the first target face to the source face shape of the second source face to obtain the second target face in the second target face image, so that the target face shape of the second target face is consistent with the source face shape of the second source face; further provides a second target face that conforms to the source face shape of the second source face for subsequent face replacement.

[0078] As an example of S302 - S303, on the basis of the example of S301 above, the face key points of face B are B i , i = 0, 1,..., 67, and the face key points of face A' are A i ＇ ; on the basis of the above example, the target center point of face B is the face center point B 30 , the target face shape key point of face B is the jaw line key point B j , j = 0, 1,..., 16, the target center point of face A' is the face center point A3 ＇ 0, the target face shape key point of face B is the jaw line key point A j ＇ ; the computer device passes through B 30 , B j , A3 ＇ 0 and A j ＇ , calculates that the face shape difference amount between face B and face A' is Diff j ; the computer device follows B j to B 30Adjustment direction, make B j and Diff j Perform pixel adjustment based on backward mapping on the target pixel points within the target adjustment area determined, and obtain the second target face image as the face image B'. In this second target face image, the second target face is the face B' in the face image B'. The target face shape of this face B' is the source face shape of the face A'.

[0079] S304: Perform face replacement on the second target face according to the second source face to obtain the replaced face image.

[0080] In the embodiments of the present application, to solve the above problems, on the basis that the aligned source face has the face image of the source face and the face pose of the target face, and the adjusted target face has the source face shape of the source face, replacing the aligned source face onto the adjusted target face to obtain the replaced face can reduce the problem of inadaptability of the face edge of the replaced face.

[0081] Based on this, after performing the above S302 - S303 to adjust the target face shape of the first target face in the first target face image to the source face shape of the second source face in the second source face image to obtain the second target face image, on the basis that the second source face in the second source face image has the face image of the first source face and the face pose of the first target face, and the second target face in the second target face image has the source face shape of the second source face, replacing the second target face with the second source face can obtain the replaced face image.

[0082] Among them, there are no obvious defects on the face edge of the replaced face in the replaced face image. The face image of the replaced face is the face image of the first source face in the first source face image, and the face pose of the replaced face is the face pose of the first target face in the first target face image.

[0083] This S304, on the basis that the target face shape of the second target face in the second target face image is consistent with the source face shape of the second source face in the second source face image, replaces the second target face with the second source face, so that there are no obvious defects on the face edge of the replaced face in the replaced face image, thereby improving the face replacement effect.

[0084] As an example of S304, on the basis of the example of the above S302 - S303, refer to Figure 5 , Figure 5 is a schematic diagram of a replaced face image provided by the embodiments of the present application; the computer device replaces the face B' in the above face image B' with the face A' in the above face image A' to obtain Figure 5 represents the replaced face image, and this replaced face image compared with Figure 1In (c), there are no major defects on the left chin and the edge of the ear of the replaced face, thus improving the face replacement effect of replacing face B with face A.

[0085] As can be seen from the above technical solution, first, the first source face in the first source face image is aligned with the first target face in the first target face image to obtain a second source face image. The face image of the second source face in the second source face image is the face image of the first source face, and the face pose of the second source face is the face pose of the first target face. This method converts the face pose of the first source face into the face pose of the first target face in the first target face image on the basis of retaining the face image of the first source face in the first source face image, so as to obtain the second source face in the second source face image, making the face image of the second source face consistent with the face image of the first source face and the face pose of the second source face consistent with the face pose of the first target face.

[0086] Then, the face shape difference between the first target face and the second source face is calculated through the target center point of the first target face, the target face shape key points, the source center point of the second source face, and the source face shape key points. According to the adjustment direction from the target face shape key points to the target center point, pixel adjustment based on backward mapping is performed on the target pixel points in the target adjustment area determined by the target face shape key points and the face shape difference amount to obtain a second target face image. The target face shape of the second target face in the second target face image is the source face shape of the second source face. This method takes into account the face shape difference between the target face shape of the first target face in the first target face image and the source face shape of the second source face in the second source face image, and adjusts the target face shape of the first target face to the source face shape of the second source face, so as to obtain the second target face in the second target face image, making the target face shape of the second target face consistent with the source face shape of the second source face.

[0087] Finally, the second target face is replaced with the second source face to obtain a replaced face image. This method replaces the second target face with the second source face in the second source face image on the basis that the target face shape of the second target face in the second target face image is consistent with the source face shape of the second source face, so that there are no obvious defects on the face edge of the replaced face in the replaced face image, thus improving the face replacement effect.

[0088] In the embodiments of the present application, when the above S303 is specifically implemented, since the pixel adjustment based on backward mapping refers to calculating the pixel value of the pixel mapped to the target pixel for the target pixel within the target adjustment area determined according to the adjustment direction from the target face key point to the target center point, so as to adjust the pixel value of the target pixel; therefore, for the pixel adjustment based on backward mapping, first, it is necessary to perform backward mapping on the target pixel within the target adjustment area determined according to the adjustment direction from the target face key point to the target center point, so as to obtain the pixel mapped to the target pixel as the mapped pixel of the target pixel; then, perform pixel resampling on the mapped pixel to obtain the resampled pixel value corresponding to the target pixel; finally, adjust the pixel value of the target pixel through the resampled pixel value corresponding to the target pixel. Based on this, the present application provides a possible implementation manner, and the above S303 includes the following S3031 - S3033 (not shown in the figure):

[0089] S3031: Perform backward mapping on the target pixel within the target adjustment area of the target face key point according to the adjustment direction from the target face key point to the target center point, so as to obtain the mapped pixel of the target pixel.

[0090] Among them, since it is considered that the surrounding pixels within at least the range of the face difference amount corresponding to the target face key point need to be pixel - adjusted, that is, the surrounding pixels within at least the range of the face difference amount corresponding to the target face key point need to be backward - mapped; therefore, it is necessary to use the target face key point as the target center, determine the target radius greater than or equal to the face difference amount according to the face difference amount, and construct a circular adjustment area through the target center and the target radius to determine the target adjustment area of the target face key point. Based on this, the present application provides a possible implementation manner. In the above S303 or S3031, the steps for determining the target adjustment area include the following S1 - S3 (not shown in the figure):

[0091] S1: Determine the target face key point as the target center.

[0092] S2: Determine the target radius according to the face difference amount; the target radius is greater than or equal to the face difference amount.

[0093] S3: Determine the target adjustment area according to the target center and the target radius.

[0094] In practical applications, in order to more simply determine a target radius greater than or equal to the face shape difference amount based on the face shape difference amount, the face shape difference amount multiplied by a preset multiple can be used as the target radius, and the preset multiple is determined according to the actual requirements of face replacement. Based on this, the present application provides a possible implementation manner, where the target radius is the face shape difference amount multiplied by a preset multiple, and the preset multiple is determined according to the actual requirements of face replacement.

[0095] Among them, on the basis that the target adjustment area has a target radius, backward mapping refers to determining, for the target pixel points within the target adjustment area constructed with the target face shape key point as the target center and the target radius determined by the face shape difference amount, the mapped pixel points that are mapped to the target pixel points according to the adjustment direction from the target face shape key point to the target center point; therefore, when performing the above S3031, the target pixel points within the target adjustment area can be backward mapped according to the adjustment direction from the target face shape key point to the target center point, through the target face shape key point, the target center point, and the target radius of the target adjustment area, to obtain the mapped pixel points of the target pixel points. Based on this, the present application provides a possible implementation manner, and the above S3031 is specifically (not shown in the figure): performing backward mapping on the target pixel points according to the adjustment direction, the target face shape key point, the target center point, and the target radius of the target adjustment area to obtain the mapped pixel points.

[0096] As an example of S3031, on the basis of the example of the above S302 - S303, refer to Figure 6 , Figure 6 FIG. is a schematic diagram of backward mapping the target pixel points within the target adjustment area to obtain the mapped pixel points according to the adjustment direction from the target face shape key point to the target center point, through the target face shape key point, the target center point, and the target radius of the target adjustment area provided by the embodiment of the present application; where point C represents the target face shape key point B j , point M represents the target center point B 30 , the circular adjustment area represents the target adjustment area, r represents the target radius of the target adjustment area, and point X represents the target pixel point; then the computer device performs backward mapping on point X within the circular adjustment area according to the adjustment direction from point C to point M, through point C, point M, and r, to obtain the mapped pixel point of point X as point U.

[0097] Among them, in the subsequent pixel adjustment process, point X within the circular adjustment area all moves along the adjustment direction from point C to point M, the closer point X is to the center of the circle, the greater the degree of movement, and the farther point X is from the center of the circle, the smaller the degree of movement, and the pixel points outside the circular adjustment area do not move.

[0098] S3032: Perform pixel resampling on the mapped pixel points to obtain the resampled pixel value corresponding to the target pixel points.

[0099] Among them, pixel resampling refers to calculating a resampled pixel value by mapping multiple pixel values corresponding to multiple neighboring pixels of a pixel point to adjust the pixel value of the target pixel point. Therefore, when performing the above S3032, the target pixel point can be pixel resampled by mapping multiple neighboring pixel points of the pixel point, and the resampled pixel value corresponding to the target pixel point can be obtained. Based on this, the present application provides a possible implementation manner, and the above S3032 is specifically (not shown in the figure): Pixel resampling is performed on the mapped pixel point according to multiple neighboring pixel points of the mapped pixel point to obtain a resampled pixel value.

[0100] As an example of S3032, the multiple nearest neighbor pixel points are N neighboring pixel points, where N is a positive integer and N≥2. On the basis of the above Figure 6 the computer device can perform pixel resampling on point U by mapping the N neighboring pixel points of point U of the pixel point, and the resampled pixel value corresponding to the target pixel point X can be obtained as RGB(x, y).

[0101] S3033: Pixel adjustment is performed on the target pixel point according to the resampled pixel value to obtain a second target face image.

[0102] Among them, when the above S3031 is specifically implemented, since the target pixel points in the target adjustment area are backward mapped according to the adjustment direction from the target face key points to the target center point, through the target face key points, the target center point, and the target radius of the target adjustment area, it actually means that on the basis of the target adjustment area determined by the target face key points and the target radius, the target face key points are moved to the target center point, so that the mapped pixel points of the target pixel points are moved to the target pixel points. Then, on the basis of the adjustment direction from the target face key points to the target center point, the target pixel points can be backward mapped through the spatial distance between the target center point and the target face key points, the spatial distance between the target pixel points and the target face key points, the target radius, and the representation vector from the target center point to the target face key points, and the mapped pixel points of the target pixel points can be obtained.

[0103] Therefore, it is necessary to calculate the spatial distance between the target center point and the target face key points in the adjustment direction as the target face distance, and calculate the spatial distance between the target pixel points and the target face key points in the adjustment direction as the pixel point distance; so as to backward map the target pixel points through the target face distance, the pixel point distance, the target radius, and the representation vector from the target center point to the target face key points to obtain the mapped pixel points of the target pixel points. Based on this, the present application provides a possible implementation manner, and the above S3031 includes the following S4 - S6 (not shown in the figure):

[0104] S4: Calculate the distances between the target center point and the target facial key points according to the adjustment direction to obtain the target facial distance.

[0105] S5: Calculate the distances between the target pixel points and the target facial key points according to the adjustment direction to obtain the pixel distances.

[0106] S6: Perform back-mapping on the target pixel points according to the target facial distance, pixel distance, target radius, and the representation vector from the target center point to the target facial key points to obtain the mapped pixel points.

[0107] As an example of S4 - S6, on the basis of the above Figure 6 the formula for obtaining the mapped pixel point U is as follows:

[0108]

[0109] where, represents the spatial distance between the target center point and the target facial key point, that is, the target facial distance; represents the spatial distance between the target pixel point and the target facial key point, that is, the pixel distance; r represents the target radius, represents the representation vector from the target center point to the target facial key point.

[0110] Among them, when specifically implementing the above S3032, since pixel resampling is performed on the mapped pixel points through multiple neighboring pixel points of the mapped pixel points, neighborhood average interpolation algorithm, nearest neighbor interpolation algorithm, bilinear interpolation algorithm, etc. can be used. Considering that the pixel adjustment of the facial image requires a high degree of authenticity, it is more appropriate to use bilinear interpolation with smoother interpolation results; therefore, first obtain multiple nearest neighbor pixel points of the mapped pixel points, and then perform interpolation processing on the multiple pixel values corresponding to the multiple nearest neighbor pixel points in the X-axis direction and Y-axis direction to obtain the resampled pixel value corresponding to the target pixel point. Based on this, the present application provides a possible implementation manner, where the multiple domain pixel points are multiple nearest neighbor pixel points, and the above S3032 includes the following S7 - S8 (not shown in the figure):

[0111] S7: Obtain multiple nearest neighbor pixel points of the mapped pixel points.

[0112] S8: Perform interpolation processing on the multiple pixel values corresponding to the multiple nearest neighbor pixel points in the X-axis direction and Y-axis direction to obtain the resampled pixel value.

[0113] As an example of this S7 - S8, on the basis of the example of the above S3032, refer to Figure 7 , Figure 7A schematic diagram for obtaining the resampled pixel value corresponding to the target pixel point by performing interpolation processing in the X-axis direction and the Y-axis direction on the multiple pixel values corresponding to the multiple nearest neighbor pixel points of the mapped pixel point; among them, points H, I, J, and K represent the 4 nearest neighbor pixel points of the mapped pixel point U corresponding to the target pixel point X. When the coordinates of point H are (i, j), the coordinates of point I are (i, j + 1), the coordinates of point J are (i + 1, j), and the coordinates of point K are (i + 1, j + 1), the calculation formula for the resampled pixel value RGB(x, y) is as follows:

[0114] RGB(E) = (x - i)(RGB(I) - RGB(H)) + RGB(H)

[0115] RGB(F) = (x - i)(RGB(K) - RGB(J)) + RGB(J)

[0116] RGB(x, y) = (y - j)(RGB(F) - RGB(E)) + RGB(E)

[0117] In the embodiment of the present application, when the above S302 is specifically implemented, based on the target face shape key points of the first target face representing the target face shape and the source face shape key points of the second source face representing the source face shape, due to the spatial distance between the target center point of the first target face and the target face shape key points of the first target face, the target face shape can be represented in the distance dimension relative to the face center point. The spatial distance between the source center point of the first source face and the source face shape key points of the first source face can represent the source face shape in the distance dimension relative to the face center point; therefore, for calculating the face shape difference amount, first calculate the spatial distance between the target center point and the target face shape key points as the target face shape distance, and calculate the spatial distance between the source center point and the source face shape key points as the source face shape distance; then calculate the distance difference amount between the target face shape distance and the source face shape distance as the face shape difference amount. Based on this, the present application provides a possible implementation manner, and the above S302 includes the following S3021 - S3023 (not shown in the figure):

[0118] S3021: Calculate the distance between the target center point and the target face shape key points to obtain the target face shape distance.

[0119] S3022: Calculate the distance between the source center point and the source face shape key points to obtain the source face shape distance.

[0120] S3023: Calculate the difference between the target face shape distance and the source face shape distance to obtain the face shape difference amount.

[0121] Among them, the target face distance is used to represent the target face shape of the first target face in the distance dimension relative to the face center point of the first target face, and the source face distance is used to represent the source face shape of the first source face in the distance dimension relative to the face center point of the first source face.

[0122] As an example of S3021 - S3023, based on the example of S302 - S303 above, the computer device first calculates the target center point B 30 and the spatial distance between the target face key point B j as the target face distance DB j , and calculates the source center point A3 ＇ 0 and the spatial distance between the source face key point A j ＇ as the source face distance DA j ＇ ; then calculates the distance difference between DB j and DA j ＇ as the face shape difference amount.

[0123] In the embodiments of the present application, when the above S301 is specifically implemented, since the face alignment model is used to retain the face image of the source face in the input source image and transform the face pose of the source face into the face pose of the target face in the input target image; and the face alignment model is used to extract multiple source face feature points of the source face and multiple target face feature points of the target face, and determine the face pose of the target face based on the multiple target face feature points to align the multiple source face feature points with the face pose of the target face; that is, the face alignment model can, by processing the face feature points, more completely and accurately transform the face pose of the source face into the face pose of the target face on the basis of retaining the face image of the source face.

[0124] Therefore, the specific implementation process of the above S301 to align the first source face in the first source face image with the first target face in the first target face image through the face alignment model to obtain the second source face image is as follows: First, the first source face image and the first target face image are input into the face alignment model, and face feature points are extracted for the first target face in the first target face image and the first source face in the first source face image, and multiple target face feature points of the first target face and multiple source face feature points of the first source face are output; then, the face pose of the first target face is determined through the multiple target face feature points; finally, the multiple source face feature points are face - aligned according to the face pose of the first target face, and the second source face image that retains both the face image of the first source face and has the face pose of the first target face is output. Based on this, the present application provides a possible implementation manner, and the above S301 includes the following S3011 - S3013 (not shown in the figure):

[0125] S3011: Extract facial feature points from the first target face and the first source face through a facial alignment model to obtain multiple target facial feature points and multiple source facial feature points.

[0126] S3012: Determine the facial pose of the first target face according to multiple target facial feature points.

[0127] S3013: Align multiple source facial feature points according to the facial pose of the first target face to obtain a second source face image.

[0128] Among them, multiple target facial feature points include multiple facial key points and multiple potential key points of the first target face, and multiple source facial feature points include multiple facial key points and multiple potential key points, and the potential key points are used to represent the facial pose.

[0129] In the embodiments of the present application, in order to improve the facial alignment efficiency and facial alignment effect of aligning the first source face in the first source face image with the first target face in the first target face image in the above S301 to obtain a second source face image; the first source face image in the above S301 can be obtained by cropping and horizontalizing the first source face from the to-be-processed source image including the first source face. Therefore, the process of obtaining the first source face image can be: First, detect the facial key points of the first source face in the to-be-processed source image to obtain multiple first facial key points; then, horizontalize the to-be-processed source image according to the multiple first facial key points to obtain a horizontalized source image including the horizontalized first source face; finally, crop the first source face in the horizontalized source image according to the multiple first facial key points to obtain the first source face image. Based on this, the present application provides a possible implementation manner. The steps of obtaining the first source face image in the above S301 include the following S9-S11 (not shown in the figure):

[0130] S9: Perform facial key point detection on the to-be-processed source image to obtain multiple first facial key points; the to-be-processed source image includes the first source face.

[0131] In practical applications, performing facial key point detection on the to-be-processed source image to obtain multiple first facial key points can actually be to perform facial key point detection on the to-be-processed source image through a facial key point detection model to obtain multiple first facial key points. Among them, the facial key point detection model can be the open-source key point detection tool dlib, and this dlib can detect 68 facial key points.

[0132] See Figure 8 , Figure 8A schematic diagram of multiple facial key points provided by an embodiment of the present application; among them, 0-67 represent multiple facial key points, 0-16 represent multiple mandibular line key points among the multiple facial key points, 17-21 represent multiple right eyebrow key points among the multiple facial key points, 22-26 represent multiple left eyebrow key points among the multiple facial key points, 27-35 represent multiple nose key points among the multiple facial key points, 36-41 represent multiple right eye key points among the multiple facial key points, 42-47 represent multiple left eye key points among the multiple facial key points, 48-60 represent multiple outer contour key points of the mouth among the multiple facial key points, and 61-67 represent multiple inner contour key points of the mouth among the multiple facial key points.

[0133] As an example of S9, the source image to be processed is Image A, and Image A includes a first source face, that is, Face A; on the basis of the above Figure 8 The computer device detects the facial key points of Face A in Image A and obtains 68 first facial key points as the first face key points.

[0134] S10: Image horizontalization is performed on the source image to be processed according to multiple first facial key points to obtain a horizontally transformed source image.

[0135] As an example of S10, on the basis of the example of S9 above, refer to Figure 9 , Figure 9 A schematic diagram of horizontally transforming the source image to be processed according to multiple first facial key points to obtain a horizontally transformed source image provided by an embodiment of the present application; among them, Figure 9 In (a), it represents the source image to be processed, which is Image A. The computer device horizontally transforms Image A according to the above 68 first face key points to obtain Figure 9 In (b), it represents the horizontally transformed source image, which includes the horizontally transformed first source face, that is, the horizontally transformed Face A.

[0136] S11: Image clipping is performed on the first source face in the horizontally transformed source image according to multiple first facial key points to obtain a first source face image.

[0137] As an example of S11, on the basis of the example of S10 above, refer to Figure 10 , Figure 10 A schematic diagram of clipping the first source face in the horizontally transformed source image according to multiple first facial key points to obtain a first source face image provided by an embodiment of the present application; among them, Figure 10 In (a), it represents the horizontally transformed source image. The computer device clips the first source face in the horizontally transformed source image, that is, Face A, according to the above 68 first face key points to obtain Figure 10 In (b), it represents the first source face image, that is, Face Image A.

[0138] Similarly, in the embodiments of the present application, the process of obtaining the first target face image may be as follows: First, detect the face key points of the first target face in the target image to be processed to obtain a plurality of second face key points; then, horizontalize the target image to be processed according to the plurality of second face key points, so that the obtained horizontalized target image includes the horizontally-aligned first target face; finally, crop the first target face in the horizontalized target image according to the plurality of second face key points to obtain the first target face image. Based on this, the present application provides a possible implementation manner. The steps of obtaining the first target face image in S301 above include the following S12-S14 (not shown in the figure):

[0139] S12: Perform face key point detection on the target image to be processed to obtain a plurality of second face key points; the target image to be processed includes the first target face.

[0140] S13: Horizontalize the target image to be processed according to the plurality of second face key points to obtain a horizontalized target image.

[0141] S14: Crop the first target face in the horizontalized target image according to the plurality of second face key points to obtain the first target face image.

[0142] Among them, considering that when there is a certain inclination angle of the face in the image, the image can be horizontalized by calculating the inclination angle and rotating the inclination angle; specifically, first, calculate the left eye center points of the plurality of left eye key points of the face and the right eye center points of the plurality of right eye key points respectively; then, calculate the included angle between the line connecting the left eye center point and the right eye center point and the horizontal direction as the face inclination angle; finally, with the center point of the connection between the left eye center point and the right eye center point as the base point, rotate the image counterclockwise by the face inclination angle to horizontalize the image. Based on this, the present application provides a possible implementation manner. The above S10 includes the following S10a-S10c (not shown in the figure):

[0143] S10a: Calculate the center point according to the plurality of first left eye key points among the plurality of first face key points to obtain the first left eye center point; calculate the center point according to the plurality of first right eye key points among the plurality of first face key points to obtain the first right eye center point.

[0144] S10b: Calculate the included angle according to the line connecting the first left eye center point and the first right eye center point and the horizontal direction to obtain the first face inclination angle.

[0145] S10c: Horizontalize the target source image to be processed according to the first face inclination angle and the first center point of the connection between the first left eye center point and the first right eye center point to obtain a horizontalized source image.

[0146] The above S13 includes the following S13a - S13c (not shown in the figure):

[0147] S13a: Calculate the center point based on multiple second left - eye key points among multiple second facial key points to obtain the second left - eye center point; calculate the center point based on multiple second right - eye key points among multiple second facial key points to obtain the second right - eye center point.

[0148] S13b: Calculate the included angle based on the line connecting the second left - eye center point and the second right - eye center point, and the horizontal direction to obtain the second facial tilt angle.

[0149] S13c: Perform image horizontalization on the target image to be processed based on the second facial tilt angle and the second center point between the second left - eye center point and the second right - eye center point to obtain the horizontally - processed target image.

[0150] Among them, considering that after image horizontalization, it is also necessary to calculate the facial position in the vertical direction and the facial position in the horizontal direction in the image to crop the face in the image. Specifically, based on the center point between the above - mentioned left - eye center point and right - eye center point, calculate the mouth center point of multiple mouth key points to locate the facial position in the vertical direction in the horizontally - processed image; and calculate the left - most key point and the right - most key point among multiple facial key points to locate the facial position in the horizontal direction in the horizontally - processed image; crop the face in the horizontally - processed image according to the facial position in the vertical direction and the facial position in the horizontal direction in the image to obtain the facial image. Based on this, the present application provides a possible implementation manner. The above S11 includes the following S11a - S11d (not shown in the figure):

[0151] S11a: Calculate the center point based on multiple mouth key points among multiple first facial key points to obtain the first mouth center point.

[0152] S11b: Perform facial positioning in the vertical direction based on the first distance between the first center point and the first mouth center point to obtain the facial position in the vertical direction in the horizontally - processed source image.

[0153] Among them, the first distance between the first center point and the first mouth center point represents the length of the middle part of the face, accounting for 40% of the overall length of the face in the vertical direction in the horizontally - processed source image. Since the length of the bottom part of the face accounts for 20% of the overall length of the face in the vertical direction in the horizontally - processed source image, and the length of the top part of the face accounts for 40% of the overall length of the face in the vertical direction in the horizontally - processed source image; then by performing facial positioning in the vertical direction based on the first distance between the first center point and the first mouth center point, the facial position in the vertical direction in the horizontally - processed source image can be obtained.

[0154] S11c: Perform face localization in the horizontal direction based on the leftmost key point and the rightmost key point among multiple first face key points, and obtain the face position in the horizontal direction in the horizontally aligned source image.

[0155] S11d: Perform image clipping on the first source face in the horizontally aligned source image according to the face position in the vertical direction and the face position in the horizontal direction in the horizontally aligned source image, and obtain the first source face image.

[0156] The above S14 includes the following S14a - S14d (not shown in the figure):

[0157] S14a: Calculate the center point based on multiple mouth key points among multiple second face key points, and obtain the second mouth center point.

[0158] S14b: Perform face localization in the vertical direction according to the second distance between the second center point and the second mouth center point, and obtain the face position in the vertical direction in the horizontally aligned target image.

[0159] Among them, the second distance between the second center point and the second mouth center point represents the length of the middle part of the face, accounting for 40% of the overall length of the face in the vertical direction in the horizontally aligned target image. Since the length of the bottom part of the face accounts for 20% of the overall length of the face in the vertical direction in the horizontally aligned target image, and the length of the top part of the face accounts for 40% of the overall length of the face in the vertical direction in the horizontally aligned target image; then by performing face localization in the vertical direction according to the second distance between the second center point and the second mouth center point, the face position in the vertical direction in the horizontally aligned target image can be obtained.

[0160] S14c: Perform face localization in the horizontal direction based on the leftmost key point and the rightmost key point among multiple second face key points, and obtain the face position in the horizontal direction in the horizontally aligned target image.

[0161] S14d: Perform image clipping on the second target face in the horizontally aligned target image according to the face position in the vertical direction and the face position in the horizontal direction in the horizontally aligned target image, and obtain the second target face image.

[0162] In summary, in the embodiments of the present application, the face replacement method can be applied to a face swapping program. Refer to Figure 11 , Figure 11 which is a schematic diagram of face replacement in a face swapping program provided by the embodiments of the present application; among them, Figure 11 in (a) represents the source face image, and the source face in the source face image is face A, Figure 11 in (b) represents the target face image, and the target face in the target face image is face B, Figure 11In figure (c), it shows the replaced face image. Even if there is a large difference in the face shapes between face A and face B, and face A cannot completely cover face B, by using the face replacement method provided in the embodiments of the present application, face A can replace face B, and there are no obvious defects in the face edges of the replaced face in the obtained replaced face image. That is, the replaced face in the replaced face image still has good face replacement quality and face replacement effect, improving the face replacement experience.

[0163] It should be noted that, based on the implementation manners provided in the above aspects of the present application, further combinations can be made to provide more implementation manners.

[0164] Based on Figure 2 the face replacement method provided in the corresponding embodiments, the embodiments of the present application further provide a face replacement device. Refer to Figure 12 , Figure 12 which is the structure diagram of a face replacement device provided in the embodiments of the present application. The face replacement device 1200 includes: an alignment unit 1201, a calculation unit 1202, an adjustment unit 1203, and a replacement unit 1204;

[0165] The alignment unit 1201 is used to perform face alignment on the first source face in the first source face image according to the first target face in the first target face image, and obtain the second source face image; the face image of the second source face in the second source face image is the face image of the first source face, and the face pose of the second source face is the face pose of the first target face;

[0166] The calculation unit 1202 is used to perform difference calculation according to the target center point of the first target face, the target face shape key points, the source center point of the second source face, and the source face shape key points, and obtain the face shape difference amount between the first target face and the second source face;

[0167] The adjustment unit 1203 is used to perform pixel adjustment based on backward mapping on the target pixel points in the target adjustment area of the target face shape key points according to the adjustment direction from the target face shape key points to the target center point, and obtain the second target face image; the target adjustment area is determined according to the target face shape key points and the face shape difference amount, and the target face shape of the second target face in the second target face image is the source face shape of the second source face;

[0168] The replacement unit 1204 is used to perform face replacement on the second target face according to the second source face, and obtain the replaced face image.

[0169] In a possible implementation manner, the adjustment unit 1203 is specifically used for:

[0170] According to the adjustment direction from the target face key points to the target center point, perform backward mapping on the target pixel points within the target adjustment area of the target face key points to obtain the mapped pixel points of the target pixel points;

[0171] Perform pixel resampling on the mapped pixel points to obtain the resampled pixel values corresponding to the target pixel points;

[0172] Perform pixel adjustment on the target pixel points according to the resampled pixel values to obtain the second target face image.

[0173] In a possible implementation manner, the adjustment unit 1203 is specifically configured to:

[0174] Perform backward mapping on the target pixel points according to the adjustment direction, the target face key points, the target center point, and the target radius of the target adjustment area to obtain the mapped pixel points; the target radius is determined according to the face shape difference amount.

[0175] In a possible implementation manner, the adjustment unit 1203 is specifically configured to:

[0176] Calculate the distance between the target center point and the target face key points according to the adjustment direction to obtain the target face distance;

[0177] Calculate the distance between the target pixel points and the target face key points according to the adjustment direction to obtain the pixel point distance;

[0178] Perform backward mapping on the target pixel points according to the target face distance, the pixel point distance, the target radius, and the representation vector from the target center point to the target face key points to obtain the mapped pixel points.

[0179] In a possible implementation manner, the adjustment unit 1203 is specifically configured to:

[0180] Perform pixel resampling on the mapped pixel points according to multiple domain pixel points of the mapped pixel points to obtain the resampled pixel values.

[0181] In a possible implementation manner, the adjustment unit 1203 is specifically configured to:

[0182] Obtain multiple nearest neighbor pixel points of the mapped pixel points;

[0183] Perform interpolation processing on the multiple pixel values corresponding to the multiple nearest neighbor pixel points in the X-axis direction and the Y-axis direction to obtain the resampled pixel values.

[0184] In a possible implementation manner, the device further includes: a determination unit;

[0185] The determination unit is specifically configured to:

[0186] Determine the target face key points as the target center of the circle;

[0187] Determine the target radius according to the face shape difference amount; the target radius is greater than or equal to the face shape difference amount;

[0188] Determine the target adjustment area according to the target center point and the target radius.

[0189] In a possible implementation, the target radius is a preset multiple of the face shape difference amount, and the preset multiple is determined according to the actual requirements of face replacement.

[0190] In a possible implementation, the calculation unit 1202 is specifically configured to:

[0191] Calculate the distance between the target center point and the target face key points to obtain the target face distance;

[0192] Calculate the distance between the source center point and the source face key points to obtain the source face distance;

[0193] Perform difference calculation according to the target face distance and the source face distance to obtain the face shape difference amount.

[0194] In a possible implementation, the alignment unit 1202 is specifically configured to:

[0195] Extract face feature points from the first target face and the first source face through a face alignment model to obtain a plurality of target face feature points and a plurality of source face feature points;

[0196] Determine the face pose of the first target face according to the plurality of target face feature points;

[0197] Align the plurality of source face feature points according to the face pose of the first target face to obtain a second source face image.

[0198] In a possible implementation, the apparatus further includes: a first obtaining unit and a second obtaining unit; the first obtaining unit is specifically configured to:

[0199] Perform face key point detection on the source image to be processed to obtain a plurality of first face key points; the source image to be processed includes the first source face;

[0200] Perform image horizontalization on the source image to be processed according to the plurality of first face key points to obtain a horizontally aligned source image;

[0201] Perform image cropping on the first source face in the horizontally aligned source image according to the plurality of first face key points to obtain a first source face image;

[0202] The second obtaining unit is specifically configured to:

[0203] Perform facial key point detection on the target image to be processed to obtain multiple second facial key points; the target image to be processed includes a first target face.

[0204] Perform image leveling on the target image to be processed according to the multiple second facial key points to obtain a leveled target image.

[0205] Perform image cropping on the first target face in the leveled target image according to the multiple second facial key points to obtain a first target face image.

[0206] It can be seen from the above technical solutions that the facial replacement device includes an alignment unit, a calculation unit, an adjustment unit, and a replacement unit. Among them, the alignment unit aligns the first source face in the first source face image with the first target face in the first target face image to obtain a second source face image. The facial image of the second source face in this second source face image is the facial image of the first source face, and the facial pose of the second source face is the facial pose of the first target face. On the basis of retaining the facial image of the first source face in the first source face image, this unit converts the facial pose of the first source face into the facial pose of the first target face in the first target face image to obtain the second source face in the second source face image, so that the facial image of the second source face is consistent with the facial image of the first source face, and the facial pose of the second source face is consistent with the facial pose of the first target face.

[0207] Among them, the calculation unit calculates the facial shape difference amount between the first target face and the second source face through the target center point of the first target face, the target facial shape key points, the source center point of the second source face, and the source facial shape key points. The adjustment unit performs pixel adjustment based on backward mapping on the target pixel points in the target adjustment area determined by the target facial shape key points and the facial shape difference amount in the adjustment direction from the target facial shape key points to the target center point to obtain a second target face image. The target facial shape of the second target face in this second target face image is the source facial shape of the second source face. This unit considers the facial shape difference between the target facial shape of the first target face in the first target face image and the source facial shape of the second source face in the second source face image, and adjusts the target facial shape of the first target face to the source facial shape of the second source face to obtain the second target face in the second target face image, so that the target facial shape of the second target face is consistent with the source facial shape of the second source face.

[0208] Among them, the replacement unit replaces the second target face with the second source face to obtain a replaced facial image. On the basis that the target facial shape of the second target face in the second target face image is consistent with the source facial shape of the second source face, this unit replaces the second target face with the second source face in the second facial image, so that there are no obvious defects in the facial edges of the replaced face in the replaced facial image, thereby improving the facial replacement effect.

[0209] The embodiment of the present application further provides a computer device, which may be a server. Refer to Figure 13 , Figure 13 FIG. is a structural diagram of a server provided by an embodiment of the present application. The server 1300 may vary greatly due to different configurations or performances, and may include one or more processors, such as the CPU 1322, and a memory 1332, and one or more storage media 1330 (such as one or more mass storage devices) for storing application programs 1342 or data 1344. Among them, the memory 1332 and the storage media 1330 may be transient storage or persistent storage. The program stored in the storage media 1330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Further, the central processing unit 1322 may be configured to communicate with the storage media 1330 and execute a series of instruction operations in the storage media 1330 on the server 1300.

[0210] The server 1300 may further include one or more power supplies 1326, one or more wired or wireless network interfaces 1350, one or more input / output interfaces 1358, and / or one or more operating systems 1341, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM , and so on.

[0211] In this embodiment, the central processing unit 922 in the server 900 may execute the methods provided in various alternative implementations of the above embodiments.

[0212] The computer device provided by the embodiment of the present application may also be a terminal. Refer to Figure 14 , Figure 14 FIG. is a structural diagram of a terminal provided by an embodiment of the present application. Taking a smart phone as an example of the terminal, the smart phone includes components such as a radio frequency (RF) circuit 1410, a memory 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a wireless fidelity (WiFi) module 1470, a processor 1480, and a power supply 1490. The input unit 1430 may include a touch panel 1431 and other input devices 1432, the display unit 1440 may include a display panel 1441, and the audio circuit 1460 may include a speaker 1461 and a microphone 1462. Those skilled in the art can understand that Figure 14The smartphone structure shown does not constitute a limitation on the smartphone, and it may include more or fewer components than those shown, or combine certain components, or have different component arrangements.

[0213] The memory 1420 can be used to store software programs and modules. The processor 1480 executes various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 1420. The memory 1420 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the smartphone (such as audio data, phone book, etc.). In addition, the memory 1420 can include high-speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0214] The processor 1480 is the control center of the smartphone, connecting various parts of the entire smartphone using various interfaces and lines. By running or executing the software programs and / or modules stored in the memory 1420, and by calling the data stored in the memory 1420, it executes various functions of the smartphone and processes data. Optionally, the processor 1480 can include one or more processing units; preferably, the processor 1480 can integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 1480.

[0215] In this embodiment, the processor 1480 in the smartphone can execute the methods provided in various optional implementation manners of the above embodiments.

[0216] According to one aspect of the present application, a computer-readable storage medium is provided. The computer-readable storage medium is used to store a computer program. When the computer program runs on a computer device, the computer device is caused to execute the methods provided in various optional implementation manners of the above embodiments.

[0217] According to one aspect of the present application, a computer program product is provided. The computer program product includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to execute the methods provided in various optional implementation manners of the above embodiments.

[0218] The descriptions of the processes or structures corresponding to the above-mentioned various drawings each have their own focuses. For parts not detailed in a certain process or structure, reference may be made to the relevant descriptions of other processes or structures.

[0219] In the description of the specification of this application and the above-mentioned drawings, terms such as "first" and "second" are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of this application described here can be implemented in an order different from those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0220] In several embodiments provided by this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of devices or units can be in electrical, mechanical or other forms.

[0221] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0222] In addition, the functional units in each embodiment of this application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0223] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device to execute all or part of the steps of the methods described in various embodiments of this application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), RAM, magnetic disks, or optical discs that can store computer programs.

[0224] As described above, the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of this application.

Claims

1. A face replacement method, characterized in that, The method includes: Aligning a first source face in a first source face image with a first target face in a first target face image to obtain a second source face image; the facial image of the second source face in the second source face image is the facial image of the first source face, and the facial pose of the second source face is the facial pose of the first target face; Calculating a difference between the first target face and the second source face in terms of facial shape by calculating the difference based on the target center point of the first target face, the target facial shape key points, the source center point of the second source face, and the source facial shape key points, to obtain the facial shape difference amount between the first target face and the second source face; Performing pixel adjustment based on backward mapping on target pixels in the target adjustment area of the target facial shape key points according to the adjustment direction from the target facial shape key points to the target center point, to obtain a second target face image; the target adjustment area is determined based on the target facial shape key points and the facial shape difference amount, and the target facial shape of the second target face in the second target face image is the source facial shape of the second source face; Replacing the second target face with the second source face to obtain a replaced face image.

2. The method according to claim 1, wherein The step of performing pixel adjustment based on backward mapping on target pixels in the target adjustment area of the target facial shape key points according to the adjustment direction from the target facial shape key points to the target center point, to obtain a second target face image, includes: Performing backward mapping on target pixels in the target adjustment area of the target facial shape key points according to the adjustment direction from the target facial shape key points to the target center point, to obtain the mapped pixels of the target pixels; Performing pixel resampling on the mapped pixels to obtain the resampled pixel values corresponding to the target pixels; Performing pixel adjustment on the target pixels according to the resampled pixel values to obtain the second target face image.

3. The method according to claim 2, wherein The step of performing backward mapping on target pixels in the target adjustment area of the target facial shape key points according to the adjustment direction from the target facial shape key points to the target center point, to obtain the mapped pixels of the target pixels, specifically is: Performing backward mapping on the target pixels according to the adjustment direction, the target facial shape key points, the target center point, and the target radius of the target adjustment area, to obtain the mapped pixels; the target radius is determined based on the facial shape difference amount.

4. The method according to claim 3, wherein The step of performing backward mapping on the target pixels according to the adjustment direction, the target facial shape key points, the target center point, and the target radius of the target adjustment area, to obtain the mapped pixels, includes: Calculating the distance between the target center point and the target facial shape key points according to the adjustment direction to obtain the target facial shape distance; Calculating the distance between the target pixels and the target facial shape key points according to the adjustment direction to obtain the pixel point distance; Performing backward mapping on the target pixels according to the target facial shape distance, the pixel point distance, the target radius, and the representation vector from the target center point to the target facial shape key points, to obtain the mapped pixels.

5. The method according to claim 2, characterized in that Performing pixel resampling on the mapped pixel points to obtain the resampled pixel value corresponding to the target pixel point specifically includes: Performing pixel resampling on the mapped pixel points according to multiple neighboring pixel points of the mapped pixel points to obtain the resampled pixel value.

6. The method according to claim 5, wherein The multiple neighboring pixel points are multiple nearest neighbor pixel points. Performing pixel resampling on the mapped pixel points according to the multiple neighboring pixel points of the mapped pixel points to obtain the resampled pixel value includes: Obtaining multiple nearest neighbor pixel points of the mapped pixel point; Performing interpolation processing on the multiple pixel values corresponding to the multiple nearest neighbor pixel points in the X-axis direction and the Y-axis direction to obtain the resampled pixel value.

7. The method according to claim 3, wherein The determining step of the target adjustment area includes: Determining the target face key point as the target center of the circle; Determining the target radius according to the face difference amount; the target radius is greater than or equal to the face difference amount; Determining the target adjustment area according to the target center of the circle and the target radius.

8. The method according to claim 7, wherein The target radius is a preset multiple of the face difference amount, and the preset multiple is determined according to the actual requirements of face replacement.

9. The method according to claim 1, wherein Performing difference calculation according to the target center point of the first target face, the target face key point, the source center point of the second source face, and the source face key point to obtain the face difference amount between the first target face and the second source face includes: Performing distance calculation on the target center point and the target face key point to obtain the target face distance; Performing distance calculation on the source center point and the source face key point to obtain the source face distance; Performing difference calculation according to the target face distance and the source face distance to obtain the face difference amount.

10. The method according to any one of claims 1-9, characterized in that, Aligning the first source face in the first source face image with the first target face in the first target face image to obtain the second source face image includes: Extracting face feature points of the first target face and the first source face through a face alignment model to obtain multiple target face feature points and multiple source face feature points; Determining the face pose of the first target face according to the multiple target face feature points; Aligning the multiple source face feature points according to the face pose of the first target face to obtain the second source face image.

11. The method according to any one of claims 1-9, characterized in that The obtaining step of the first source face image includes: Performing face key point detection on the source image to be processed to obtain multiple first face key points; the source image to be processed includes the first source face; Performing image leveling on the source image to be processed according to the multiple first face key points to obtain a leveled source image; Performing image cropping on the first source face in the leveled source image according to the multiple first face key points to obtain the first source face image; The obtaining step of the first target face image includes: Performing face key point detection on the target image to be processed to obtain multiple second face key points; the target image to be processed includes the first target face; Performing image leveling on the target image to be processed according to the multiple second face key points to obtain a leveled target image; Perform image clipping on the first target face in the leveled target image according to the multiple second facial key points to obtain the first target face image.

12. A face replacement device, characterized in that, The device includes: an alignment unit, a calculation unit, an adjustment unit, and a replacement unit; The alignment unit is configured to perform facial alignment on the first source face in the first source face image according to the first target face in the first target face image to obtain a second source face image; the facial image of the second source face in the second source face image is the facial image of the first source face, and the facial pose of the second source face is the facial pose of the first target face; The calculation unit is configured to perform difference calculation according to the target center point of the first target face, the target facial shape key points, the source center point of the second source face, and the source facial shape key points to obtain the facial shape difference amount between the first target face and the second source face; The adjustment unit is configured to perform pixel adjustment based on backward mapping on the target pixel points in the target adjustment area of the target facial shape key points according to the adjustment direction from the target facial shape key points to the target center point to obtain a second target face image; the target adjustment area is determined according to the target facial shape key points and the facial shape difference amount, and the target facial shape of the second target face in the second target face image is the source facial shape of the second source face; The replacement unit is configured to perform facial replacement on the second target face according to the second source face to obtain a replaced facial image.

13. A computer device, characterized in that, The computer device includes a processor and a memory: The memory is configured to store a computer program and transmit the computer program to the processor; The processor is configured to execute the method according to any one of claims 1-11 according to the instructions in the computer program.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium is configured to store a computer program, and when the computer program runs on a computer device, the computer device is caused to execute the method according to any one of claims 1-11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program runs on a computer device, the computer device is caused to execute the method according to any one of claims 1-11.

Images