Face image processing method and device, computer equipment and storage medium

Abstract

The invention relates to a face image processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring two-dimensional image information of a reference face and two-dimensional image information of a target face; based on the two-dimensional image information, performing three-dimensional reconstruction on the reference face and the target face to obtain respective three-dimensional face models of the reference face and the target face; determining a deformation field of the target face relative to the reference face according to respective mapping positions of the face feature points in a selected two-dimensional plane of each three-dimensional face model; and adjusting the two-dimensional image information of the target face according to the deformation field to obtain an updated face image of the target face. Wherein the selected two-dimensional planes of different three-dimensional face models correspond to the same face angle. By adopting the method, the updated face image which can fit the template effect of the reference face and can reflect the features of the target face can be obtained, and the effect of face image processing can be further improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a facial image processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of computer technology, facial image processing technology based on computer vision has emerged. Through facial image processing, facial images can be beautified and stylized. For example, the shape of the face can be changed to make it look more delicate or cute.

[0003] Traditional techniques adjust facial images based on standard deformation fields corresponding to fixed deformation templates, such as "thin face" and "wide eyes." Using a single template makes it difficult to achieve diverse facial editing effects. Furthermore, using multiple templates to superimpose deformations can lead to the loss of original facial features. Consequently, traditional facial image processing methods suffer from poor image processing performance. Summary of the Invention

[0004] Based on this, it is necessary to provide a face image processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve the image processing effect in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for processing facial images. The method comprises:

[0006] Obtaining two-dimensional image information of the reference face and the target face;

[0007] Based on the two-dimensional image information, the reference face and the target face are respectively reconstructed in three dimensions to obtain three-dimensional face models of the reference face and the target face;

[0008] determining a deformation field of the target face relative to the reference face based on respective mapping positions of facial feature points in the selected two-dimensional planes of each of the three-dimensional face models; the selected two-dimensional planes of different three-dimensional face models correspond to the same facial angle;

[0009] The two-dimensional image information of the target face is adjusted according to the deformation field to obtain an updated facial image of the target face.

[0010] In a second aspect, the present application further provides a facial image processing device. The device comprises:

[0011] An image information acquisition module is used to acquire two-dimensional image information of the reference face and the target face;

[0012] a three-dimensional reconstruction module, configured to perform three-dimensional reconstruction on the reference face and the target face based on the two-dimensional image information, to obtain three-dimensional face models of the reference face and the target face respectively;

[0013] a deformation field determination module, configured to determine the deformation field of the target face relative to the reference face based on the respective mapping positions of the facial feature points in the selected two-dimensional planes of each of the three-dimensional face models; the selected two-dimensional planes of different three-dimensional face models correspond to the same facial angle;

[0014] An image updating module is used to adjust the two-dimensional image information of the target face according to the deformation field to obtain an updated facial image of the target face.

[0015] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0016] Obtaining two-dimensional image information of the reference face and the target face;

[0017] Based on the two-dimensional image information, the reference face and the target face are respectively reconstructed in three dimensions to obtain three-dimensional face models of the reference face and the target face;

[0018] determining a deformation field of the target face relative to the reference face based on respective mapping positions of facial feature points in the selected two-dimensional planes of each of the three-dimensional face models; the selected two-dimensional planes of different three-dimensional face models correspond to the same facial angle;

[0019] The two-dimensional image information of the target face is adjusted according to the deformation field to obtain an updated facial image of the target face.

[0020] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0021] Obtaining two-dimensional image information of the reference face and the target face;

[0022] Based on the two-dimensional image information, the reference face and the target face are respectively reconstructed in three dimensions to obtain three-dimensional face models of the reference face and the target face;

[0023] determining a deformation field of the target face relative to the reference face based on respective mapping positions of facial feature points in the selected two-dimensional planes of each of the three-dimensional face models; the selected two-dimensional planes of different three-dimensional face models correspond to the same facial angle;

[0024] The two-dimensional image information of the target face is adjusted according to the deformation field to obtain an updated facial image of the target face.

[0025] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0026] Obtaining two-dimensional image information of the reference face and the target face;

[0027] Based on the two-dimensional image information, the reference face and the target face are respectively reconstructed in three dimensions to obtain three-dimensional face models of the reference face and the target face;

[0028] determining a deformation field of the target face relative to the reference face based on respective mapping positions of facial feature points in the selected two-dimensional planes of each of the three-dimensional face models; the selected two-dimensional planes of different three-dimensional face models correspond to the same facial angle;

[0029] The two-dimensional image information of the target face is adjusted according to the deformation field to obtain an updated facial image of the target face.

[0030] The above-mentioned facial image processing method, device, computer equipment, computer-readable storage medium and computer program product, through a transformation and alignment method from two-dimensional to three-dimensional and then to two-dimensional, can obtain a deformation field of the target face relative to the reference face without being affected by the posture of the reference face and the target face, which is conducive to improving the flexibility of facial image processing and enhancing the facial image processing effect; and, based on the mapping position of the facial feature points, the deformation field is determined so that the obtained deformation field conforms to the facial features of the target face, thereby ensuring that an updated facial image can be obtained based on the deformation field, which can not only fit the template effect of the reference face but also reflect the features of the target face, which is conducive to further improving the effect of facial image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A diagram showing an application environment of a face image processing method according to an embodiment;

[0032] Figure 2 Schematic diagram of the implementation process of a face image processing method in one embodiment;

[0033] Figure 3 1 is a flow chart of a facial image processing method according to an embodiment;

[0034] Figure 4 Schematic diagram of the mapping positions of facial feature points in a two-dimensional selected plane in one embodiment;

[0035] Figure 5 FIG1 is a schematic diagram of an alignment process of a reference face and a target face in one embodiment;

[0036] Figure 6 is a flowchart of a face image processing method according to another embodiment;

[0037] Figure 7 A schematic diagram of the deformation principle of traditional face image processing in one embodiment;

[0038] Figure 8 A schematic diagram of deformation effects of traditional face image processing in one embodiment;

[0039] Figure 9 Schematic diagram of the implementation process of a face image processing method in another embodiment;

[0040] Figure 10 is a structural block diagram of a face image processing device in one embodiment;

[0041] Figure 11 is a diagram of the internal structure of a computer device in one embodiment;

[0042] Figure 12 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0044] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0045] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0046] Computer vision (CV) is the science of enabling machines to "see." Specifically, it refers to machine vision techniques such as using cameras and computers to replace the human eye in identifying and measuring objects, and then performing further image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the field of vision, such as the Swin Transformer, ViT, V-MOE, and MAE, can be fine-tuned to quickly and widely apply to specific downstream tasks. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0047] The solutions provided in the embodiments of this application involve technologies such as artificial intelligence face image processing, and are specifically described through the following embodiments:

[0048] The face image processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. The communication network can be a wired network or a wireless network. Therefore, terminal 102 and server 104 can be directly or indirectly connected via wired or wireless communication. For example, terminal 102 can be indirectly connected to server 104 via a wireless access point, or terminal 102 can be directly connected to server 104 via the Internet, which is not limited in this application.

[0049] Terminal 102 may include, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices may include smart watches, smart bracelets, head-mounted devices, etc. The embodiments of the present application can be applied to face beautification or face stylization conversion scenarios. Terminal 102 may be installed with a client related to facial image processing. This client may include separately installed applications such as browsers, image acquisition clients, video editing clients, etc., or may include mini-programs or web pages that can be used without downloading. Server 104 is a backend server corresponding to the client, or a server dedicated to facial image processing. Furthermore, server 104 may be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The data storage system can store data that needs to be processed by the server 104. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers.

[0050] It should be noted that the face image processing method in the embodiment of the present application can be executed by the terminal 102 or the server 104 alone, or can be executed by the terminal 102 and the server 104 together. Figure 2 As shown, during facial image processing, server 104 can obtain 2D image information of a reference face and a target face. Based on the 2D image information, server 104 can perform 3D reconstruction of the reference face and the target face, respectively, to obtain 3D facial models of the reference face and the target face. Next, server 104 performs 2D point mapping and 2D face alignment based on facial feature points. Specifically, server 104 determines the deformation field of the target face relative to the reference face based on the mapping positions of the facial feature points in the selected 2D planes of each 3D facial model. Finally, server 104 adjusts the 2D image information of the target face based on the deformation field to obtain an updated facial image of the target face. The selected 2D planes of different 3D facial models correspond to the same facial angle. The 2D image information can be selected or uploaded by the user via terminal 102.

[0051] In one embodiment, Figure 3 As shown, a face image processing method is provided, which can be executed by a computer device. The computer device can be Figure 1The terminal or server shown in this embodiment is applied to Figure 1 The following steps are used as an example to illustrate the server in the example:

[0052] Step S302: Acquire two-dimensional image information of the reference face and the target face.

[0053] The reference face is the face used as a reference, that is, the face used as a deformation template during facial image processing. This reference face can be, for example, a face with certain stylistic characteristics, such as stylized faces from different cartoon characters, or a facial template that meets aesthetic requirements. The target face is the face that requires facial image adjustment, that is, the face to be processed during facial image processing. A two-dimensional image is a flat image that does not contain depth information.

[0054] The two-dimensional image is presented in a non-quantized form and can be stored in the form of an image file on an electronic computer, such as jpeg, bmp, etc. Furthermore, the two-dimensional image information of the reference face can be a two-dimensional face image obtained from a local location or the Internet, or a two-dimensional face image obtained by collection. The two-dimensional image information of the target face can be a two-dimensional face image captured in real time by a camera, or a two-dimensional face image obtained from a local location or the Internet. This embodiment does not limit the facial posture and expression corresponding to the two-dimensional image information. In addition, the two-dimensional image information of the face can be a single two-dimensional image frame, or it can include multiple two-dimensional image frames, that is, this embodiment does not limit the number of image frames contained in the two-dimensional image information.

[0055] Specifically, the server can obtain the two-dimensional image information of the reference face and the target face. The specific way in which the server obtains the two-dimensional image information can be active acquisition or passive reception, which is not limited here.

[0056] In a specific embodiment, a user may upload image information containing a facial image via a terminal. The server may then perform facial image recognition on the image information, extract an image region containing the facial image from the image information, and then obtain two-dimensional image information of the corresponding face from the image region. The facial image may, for example, be a facial image of a reference face, a facial image of a target face, or may include facial images of both the reference face and the target face.

[0057] In a specific embodiment, a user can upload a facial image of a reference face to a server through a terminal, and the server can then obtain two-dimensional image information of the reference face from the facial image as a facial template in the facial image processing process; the server can also obtain facial images of real faces collected in real time by the terminal to obtain two-dimensional image information of the target face.

[0058] Step S304 : Based on the two-dimensional image information, the reference face and the target face are respectively reconstructed in three dimensions to obtain three-dimensional face models of the reference face and the target face.

[0059] 3D reconstruction refers to the creation of mathematical models suitable for computer representation and processing of 3D objects. It serves as the basis for processing, manipulating, and analyzing the properties of 3D objects in a computer environment. It is also a key technology for creating virtual reality representations of the objective world within computers. Specifically for this application, 3D reconstruction of a human face is the process of obtaining a 3D face model.

[0060] Specifically, the server can perform 3D reconstruction of the reference face based on the 2D image information of the reference face to obtain a 3D facial model of the reference face; and perform 3D reconstruction of the target face based on the 2D image information to obtain a 3D facial model of the target face. There is no single specific method for performing 3D reconstruction. Alternatively, the server can perform 3D reconstruction based on the 2D image information to obtain a 3D facial model based on the 2D image information using any of three-dimensional modeling techniques (3DMM), deep learning-based prediction schemes, or back-projection reconstruction algorithms. Taking the back-projection reconstruction algorithm as an example, the server can perform face detection and key point recognition on the 2D facial image information to obtain 2D facial key points. Then, using error constraints imposed by back-projecting the 2D key points back into pixel space, the server can perform 3D reconstruction of the face to obtain a 3D facial model.

[0061] In one specific embodiment, the two-dimensional image information of the target face includes multiple two-dimensional image frames. In this embodiment, the process of obtaining a three-dimensional face model of the target face includes: performing facial pose recognition on each two-dimensional image frame to determine the facial pose of each two-dimensional image frame; determining the depth information of the facial key points based on the facial poses and the respective position information of the same facial key points in each two-dimensional image frame; and performing three-dimensional reconstruction of the target face based on the respective position information and depth information of the multiple facial key points to obtain a three-dimensional face model of the target face.

[0062] Among them, facial key points refer to points on the face that can represent key information. The facial key points may include key points of the eyes, eyebrows, mouth, nose, and facial contours. Furthermore, facial key points can be pre-defined or determined by measurement. Facial key point detection algorithms include but are not limited to model-based ASM (Active Shape Model) and AAM (Active Appearance Model), cascaded shape regression CPR (Cascaded Pose Regression) algorithms, and deep learning-based algorithms, etc., which are not limited here. Facial posture can be represented by the relative position information between the face and the camera device in the facial image. Facial posture includes but is not limited to angle posture and distance posture. Angle posture refers to the angular position between the face and the camera device in the facial image to be identified. Distance posture refers to the distance position between the face and the camera device in the facial image to be identified.

[0063] Specifically, the server can perform facial recognition on each 2D image frame to determine the facial region in each 2D image frame. Then, using a pre-trained facial pose recognition model, the server can perform facial pose recognition on each facial region to determine the facial pose in each 2D image frame. Next, the server can determine the depth information of each facial key point based on its position information in each 2D image frame and the facial pose in each 2D image frame. Finally, based on the position information and depth information of the multiple facial key points, the server can perform a 3D reconstruction of the target face to obtain a 3D facial model of the target face.

[0064] In one possible implementation, the server can select one of the facial key points as a reference point. For each facial key point, the server then determines the depth of the facial key point relative to the reference point, combining the position information of the facial key point in each 2D image frame and the facial pose of each 2D image frame. This allows the server to obtain the relative position and depth of each facial key point. It will be appreciated that once the relative positions and depths are obtained, the position and depth information of each facial key point can be determined, providing the basis for 3D reconstruction.

[0065] It should be noted that the reconstruction algorithms used in the 3D reconstruction process of the reference face and the target face can be the same or different. For example, a user can upload a 2D image of the reference face and multiple 2D image frames containing the target face through a terminal. The server can then use the 3DMM algorithm to convert the 2D image information into 3D information to obtain a 3D face model of the reference face. Facial pose recognition can be performed on each 2D image frame to determine the facial pose of each 2D image frame. Based on the facial pose and the position information of the same facial key points in each 2D image frame, the depth information of the facial key points can be determined. Finally, based on the position information and depth information of the multiple facial key points, the target face can be 3D reconstructed to obtain a 3D face model of the target face.

[0066] In the above embodiment, facial posture recognition is performed on multiple two-dimensional image frames of the face, the position information and depth information of multiple facial key points are determined, and then three-dimensional reconstruction of the face is completed to obtain a three-dimensional face model. This can ensure the accuracy of the three-dimensional face model and thus ensure the effect of facial image processing based on the three-dimensional face model.

[0067] Step S306 : determining the deformation field of the target face relative to the reference face based on the mapping positions of the facial feature points in the selected two-dimensional plane of each three-dimensional face model.

[0068] Among them, facial feature points refer to points that can reflect facial features. The facial feature points can specifically be points that can reflect the distribution of facial features, such as the eyes, eyebrows, nose, mouth, cheeks, etc. For example, eye feature points can specifically include feature points of eyelids, irises, etc.; nose feature points can specifically include feature points of nose tip, nose root, etc. For the same face, the facial pixels contained in facial feature points and facial key points can be the same, partially the same, or completely different. For example, a part of the pixels at the eyelid position can be used as facial key points, and another part of the pixels can be used as facial feature points. In a specific implementation, the same pixel on the face can be used as both a facial feature point and a facial key point, thereby reducing the amount of computation in the data processing process and improving efficiency.

[0069] Furthermore, the selected two-dimensional planes of different three-dimensional face models correspond to the same facial angle. In the three-dimensional space where the three-dimensional face model is located, the two-dimensional plane can be represented by the facial angle of the face. The facial angle can be represented by the facial posture angle, which can specifically include the facial pitch angle (the angle of the face rotating around the X-axis), the facial yaw angle (the angle of the face rotating around the Y-axis), and the facial roll angle (the angle of the face rotating around the Z-axis). Among them, the vertical plane of the Z-axis is the two-dimensional plane corresponding to the facial angle of the frontal posture. The field is the distribution of objects in space. The deformation field of the target face relative to the reference face refers to the offset from the target face image to the reference face image, which can specifically include the transformation direction and displacement.

[0070] It is understandable that due to the existence of perspective relationships, the relative positions of facial feature points (e.g., facial features) on the face differ at different facial angles. Based on this, the server can first determine the selected two-dimensional planes representing the same facial angle in the three-dimensional space where the different three-dimensional face models are located. In a specific embodiment, the two-dimensional plane represented by the facial angle of the frontal posture can be used as the selected two-dimensional plane to ensure that the resulting deformation field can reflect the deviation in the distribution of feature points of the target face relative to the reference face. In a specific embodiment, the two-dimensional plane represented by the facial angle of the target face in the two-dimensional image information can be used as the selected two-dimensional plane to facilitate subsequent two-dimensional image adjustments.

[0071] After determining the selected 2D plane, the server maps the facial feature points from different 3D face models onto the corresponding 2D planes, obtaining the corresponding mapping positions of the facial feature points in each 3D face model. This allows the server to determine the difference in mapping position of the target face relative to the reference face. Finally, based on the difference in mapping position of each facial feature point, the server determines the deformation field of the target face relative to the reference face.

[0072] In a specific implementation, the 3D face model ReConstruction (F m ) After that, we can get the pose of the reference face and the face morphology fitting result F in three-dimensional space. M3d , by changing F M3d Project the cluster along the Z axis, that is, F M3d Mapped to a Z plane space, a set of serial numbers and two-dimensional coordinate points KP corresponding to the two-dimensional features are obtained. M :

[0073] pose,F M3d =ReConstruction(F m )

[0074] KP M=Proj(F M3d )

[0075] For the two-dimensional image information of the target face, after the same processing, the Z plane projection point KP of the target face can be obtained r Then, after the key point matching and alignment mapping operation of the eye, we can get the KP M Corresponding points KP in the same plane space r ', the deformation field Diff_xy can be obtained by KP M With KP r 'The deviation between them is expressed as: Diff_xy = (KP r '_x- KP M _x ,KP r '_y- KP M _y).

[0076] Step S308 : adjusting the two-dimensional image information of the target face according to the deformation field to obtain an updated facial image of the target face.

[0077] Specifically, after obtaining the deformation field, the two-dimensional image information of the target face can be adjusted based on the deformation field to obtain an updated facial image of the target face. Taking the case where the two-dimensional image information of the target face includes multiple two-dimensional image frames as an example, in a specific embodiment, the server can perform two-dimensional mapping based on the facial angle of the target facial image in each two-dimensional image frame of the target face, determine the deformation field of the target face relative to the reference face at that facial angle, and adjust the image information of the target face in the two-dimensional image frame based on the deformation field to obtain an updated facial image corresponding to the two-dimensional image frame.

[0078] In a specific embodiment, the server may first obtain the deformation field Diff_xy corresponding to the frontal face posture, and then, for each two-dimensional image frame, perform a certain correction on the deformation field Diff_xy according to the facial angle theta and the face size scale of the target face in the two-dimensional image frame to obtain a corrected deformation field Diff_xy' that better fits the target face:

[0079] Diff_xy'=Correction(Diff_xy,theta,scale)

[0080] Then, according to the offset of each pixel point represented by the correction deformation field, the two-dimensional image of the target face in the two-dimensional image frame is adjusted to obtain an updated facial image of the target face.

[0081] There is no single way to correct the deformation field. For example, the server can determine the perspective effect under the angular deviation between the facial angle corresponding to the deformation field and the facial angle of the target face in the two-dimensional image information, and then determine the correction coefficient of the deformation field at different positions on the face. The correction coefficient can specifically be a coefficient for stretching or compressing the deformation field. The deformation field is then corrected based on the correction coefficient to obtain a corrected deformation field. For another example, the server can also determine the coordinate transformation coefficients of the facial feature points in the two-dimensional image information of the target face based on the respective facial sizes of the target face and the reference face, and then correct the offset of the facial feature points based on the coordinate transformation coefficients to obtain the corrected offset, and then obtain the corrected deformation field corresponding to each corrected offset.

[0082] In other words, using the above-mentioned image processing method for image adjustment can adjust the target facial image in a direction close to the reference facial image, but will maintain the facial features of the target face to a certain extent, which is equivalent to generating an updated facial image exclusive to the target face.

[0083] The facial image processing method described above obtains 2D image information of a reference face and a target face; based on the 2D image information, performs 3D reconstruction on the reference face and the target face to obtain 3D facial models of the reference face and the target face, respectively; determines the deformation field of the target face relative to the reference face based on the mapping positions of facial feature points in a selected 2D plane of each 3D facial model; and adjusts the 2D image information of the target face based on the deformation field to obtain an updated facial image of the target face. The selected 2D planes of different 3D facial models correspond to the same facial angle. In the above-mentioned facial image processing process, the transformation and alignment method from two-dimensional to three-dimensional and then to two-dimensional can obtain the deformation field of the target face relative to the reference face without being affected by the postures of the reference face and the target face, which is conducive to improving the flexibility of facial image processing and improving the facial image processing effect; and, based on the mapping position of the facial feature points, the deformation field is determined so that the obtained deformation field conforms to the facial features of the target face, thereby ensuring that an updated facial image can be obtained according to the deformation field, which can not only fit the template effect of the reference face but also reflect the features of the target face, which is conducive to further improving the effect of facial image processing.

[0084] In one embodiment, step S304 includes: based on each two-dimensional image information, performing expression recognition on the reference face and the target face respectively, and determining the expression features of the reference face and the target face respectively; when the similarity conditions are not met between the expression features, replacing the expression of at least one of the reference face and the target face to obtain updated image information; based on each updated image information, performing three-dimensional reconstruction on the reference face and the target face respectively, and obtaining a three-dimensional face model of the reference face and the target face respectively.

[0085] Here, a representational feature refers to characteristic information that reflects the characteristics of a facial expression. The specific data form of the expression feature can be a vector or a matrix. The fact that the expression features do not meet the similarity condition means that the similarity between the expression features is less than or equal to the similarity threshold. The purpose of performing expression replacement to obtain updated image information is to make the expressions of the reference face and the target face more consistent. In other words, after expression replacement, the expression features corresponding to each updated image information meet the similarity condition.

[0086] Specifically, the server can perform expression recognition on the reference face and the target face based on the two-dimensional image information to determine the expression features of each reference face and the target face. Alternatively, the server can perform expression recognition on the reference face and the target face based on a pre-trained expression feature extraction model to determine the expression features of each reference face and the target face. The server can also use optical flow to determine the main direction of muscle movement and then extract the optical flow value in the local space to form an expression feature vector.

[0087] Because the positions of facial feature points vary significantly under different expressions, if a 3D reconstruction of a face is performed to obtain a 3D face model when the labels corresponding to the 2D image information differ significantly, the subsequent differences in the mapped positions obtained based on this reconstruction will be affected to some extent by the expression factor, and may not accurately reflect the differences between the target face and the reference face. Based on this, after obtaining the expression features of each reference face and the target face, the server can calculate the similarity between each expression feature. If the similarity condition is met between each expression feature, it indicates that the original expressions of the reference face and the target face in their respective 2D image information are relatively similar. In this case, the server can directly perform 3D reconstruction based on each 2D image information to obtain a 3D face model. If the similarity condition is not met between each expression feature, it indicates that the original expressions of the reference face and the target face in their respective 2D image information are significantly different. In this case, the server can replace the expression of at least one of the reference face and the target face to obtain updated image information. Based on each updated image information, the server can perform 3D reconstruction on the reference face and the target face, respectively, to obtain a 3D face model for each reference face and the target face.

[0088] In the above embodiment, when the expression features do not meet the similarity condition, updated image information is obtained by expression replacement, and then three-dimensional reconstruction is performed based on the updated image information. This can eliminate the influence of expression on three-dimensional modeling and improve the accuracy of the three-dimensional face model.

[0089] It is understandable that the specific manner of performing expression replacement on at least one of the reference face and the target face is not unique.

[0090] In a specific embodiment, the expression replacement may be performed on only one of the reference face and the target face. Specifically, the server may replace the facial expression of the reference face with the facial expression of the target face, or replace the facial expression of the target face with the facial expression of the reference face in the two-dimensional image information, to obtain updated image information with a consistent expression.

[0091] Furthermore, expression replacement can also be performed on both the reference face and the target face. In one embodiment, if the expression features do not meet a similarity condition, expression replacement is performed on at least one of the reference face and the target face to obtain updated image information, including: obtaining standard expression features corresponding to a standard expression image; and for each two-dimensional image, if the expression features corresponding to the two-dimensional image information do not meet a similarity condition with the standard expression features, performing expression replacement on the face represented by the two-dimensional image information based on the standard expression image to obtain updated image information of the face.

[0092] The standard expression image may be a standard expressionless image, a standard smiling expression image, or the like. Specifically, the server may obtain the standard expression features corresponding to the standard expression image from an expression database. Then, for each two-dimensional image, the server calculates the similarity between the expression features corresponding to the two-dimensional image and the standard expression features. If the similarity does not meet the similarity condition, the server replaces the expression of the face represented by the two-dimensional image based on the standard expression image to obtain updated image information of the face. In a specific implementation, the standard expression image may be a standard expressionless image. That is, the server may obtain updated expressionless image information by removing the expression from the two-dimensional image information of the reference face and the target face.

[0093] In the above embodiment, expression replacement is performed based on the standard expression image to obtain updated image information of the face, which can ensure that the updated image information is not affected by the original expression and improve the accuracy of the three-dimensional face model obtained by three-dimensional reconstruction based on the updated image information.

[0094] In one embodiment, the deformation field of the target face relative to the reference face is determined based on the respective mapping positions of the facial feature points in the selected two-dimensional planes of each three-dimensional face model, including: determining the selected two-dimensional planes represented by the selected facial angles in the three-dimensional space where each three-dimensional face model is located; and determining the deformation field of the target face relative to the reference face based on the respective mapping positions of the facial feature points in each selected two-dimensional plane.

[0095] In practice, due to perspective, the relative positions of facial feature points differ at different facial angles. Mapping feature points based on the selected 2D planes representing these different facial angles will not accurately reflect the feature point deviations between the target and reference faces. To address this, the server can first determine the selected 2D planes representing the same selected facial angle within the 3D space of each 3D face model.

[0096] After determining the selected 2D plane, the server maps the facial feature points from different 3D face models onto the corresponding 2D planes, obtaining the corresponding mapping positions of the facial feature points in each 3D face model. This allows the server to determine the difference in mapping position of the target face relative to the reference face. Finally, based on the difference in mapping position of each facial feature point, the server determines the deformation field of the target face relative to the reference face.

[0097] In the above embodiment, the selected two-dimensional plane represented by the selected facial angle is first determined, and then the feature points are mapped. This can ensure that the obtained deformation field can truly reflect the feature point distribution deviation of the target face relative to the reference face, and further ensure that image adjustment based on the deformation field can obtain better image processing effects.

[0098] In practical applications, different facial angles can be selected as the selected facial angle. For example, the selected facial angle can be a facial angle of a frontal face pose, or a facial angle of a reference face or a target face in two-dimensional image information.

[0099] In a specific embodiment, the facial image processing method further includes: performing facial angle recognition on the target face based on two-dimensional image information of the target face to determine the facial angle of the target face; and determining the facial angle of the target face as the selected facial angle.

[0100] Specifically, the server can perform facial angle recognition on the target face based on the two-dimensional image information of the target face, determine the facial angle of the target face in the two-dimensional image information, and determine the facial angle as the selected facial angle. There is no single specific method for performing facial angle recognition. For example, the server can use a pre-trained facial pose network (pose head) to perform pose estimation on the two-dimensional image information of the target face to determine the facial angle of the target face. In another example, the server can perform feature extraction processing on the two-dimensional image information of the target face using a feature extraction network to obtain facial angle features of the target face. The server can then perform feature conversion processing on the facial angle features using a pre-set feature conversion network to obtain a rotation matrix of the target face. Based on the rotation matrix, the server can determine the facial pose angle of the target face, i.e., the facial angle of the target face in the two-dimensional image information.

[0101] It should be noted that when the two-dimensional image information includes multiple two-dimensional image frames, the server can perform facial angle recognition and feature point mapping on each two-dimensional image frame to obtain a deformation field corresponding to each two-dimensional image frame. Based on the deformation field corresponding to each two-dimensional image frame, the server can adjust the facial image information of each two-dimensional image frame to obtain an updated facial image corresponding to each two-dimensional image frame. The server can also determine a selected image frame from each two-dimensional image frame, perform facial angle recognition and feature point mapping on the selected image frame to obtain a deformation field corresponding to the selected image frame, correct the deformation field based on the facial angle differences between each non-selected image frame and the selected image frame, thereby obtaining a corrected deformation field corresponding to each non-selected image frame. The server can then adjust the facial information of each image frame based on the corresponding deformation field to obtain an updated facial image corresponding to each two-dimensional image frame. The selected image frame can refer to a two-dimensional image frame whose facial pose angle meets the facial recognition requirements. Generally speaking, facial pose angles within ±30 degrees meet the facial recognition requirements.

[0102] In the above embodiment, the facial angle of the target face in the two-dimensional image information is determined as the selected facial angle, which can ensure the matching degree between the two-dimensional image information and the deformation field, thereby ensuring that the two-dimensional image information is adjusted based on the deformation field, and an updated facial image reflecting the characteristics of the target face can be obtained, which is conducive to further improving the image processing effect.

[0103] In one specific embodiment, the facial image processing method further includes determining the facial angle corresponding to the frontal facial posture as the selected facial angle. In this embodiment, step S308 includes determining the angular deviation between the facial angle of the target face in the two-dimensional image information and the selected facial angle; correcting the deformation field based on the angular deviation to obtain a corrected deformation field; and adjusting the two-dimensional image information of the target face based on the corrected deformation field to obtain an updated facial image of the target face.

[0104] Among them, the facial angle corresponding to the frontal face posture is: the face pitch angle, face yaw angle, and face roll angle are all zero. Specifically, the server can determine the facial angle corresponding to the frontal face posture as the selected facial angle. Then, the server determines the angular deviation between the facial angle of the target face in the two-dimensional image information and the selected facial angle. The angular deviation can specifically include the pitch angle deviation, the yaw angle deviation, and the roll angle deviation. After determining the angular deviation, the server can correct the deformation field according to the spatial relationship perspective information corresponding to the angular deviation to obtain a corrected deformation field, and adjust the two-dimensional image information of the target face according to the corrected deformation field to obtain an updated facial image of the target face.

[0105] In the above embodiment, the facial angle corresponding to the frontal face posture is determined as the selected facial angle. On the one hand, it can ensure that the deformation field fully reflects the facial deformation of the target face relative to the reference face. On this basis, the deformation field is corrected according to the angle deviation to obtain a corrected deformation field, and image adjustment is performed based on the corrected deformation field. A more accurate updated facial image can be obtained, which is conducive to further improving the image processing effect. On the other hand, it is equivalent to only performing feature point mapping for the frontal face posture, which can reduce the workload of facial image processing and improve work efficiency.

[0106] In one embodiment, the facial image processing method further includes determining a reference facial size of a reference face and a target facial size of a target face. In this embodiment, correcting the deformation field based on the angular deviation to obtain a corrected deformation field includes correcting the deformation field based on the angular deviation and the size deviation between the reference facial size and the target facial size to obtain the corrected deformation field.

[0107] Facial dimensions can include the outer contour dimensions of the face, the distance between the eyes, the length of the nose bridge, the size of the eyes, the length of the philtrum, and so on. Based on the angular deviation, the deformation field can also be corrected in combination with the facial dimensions. Specifically, the server can determine the reference facial dimensions of the reference face and the target facial dimensions of the target face, calculate the size deviation between the reference and target facial dimensions, and correct the deformation field based on the angular deviation and size deviation to obtain a corrected deformation field. For example, if the facial dimensions are the outer contour dimensions, the target face with a larger outer contour dimension will have a larger feature point offset than a target face with a smaller outer contour dimension. To preserve the features of the target face in terms of the outer contour dimension, the server can appropriately reduce the feature point offset for the larger target face, thereby obtaining a corrected deformation field by stretching the deformation field. For the smaller target face, the server can appropriately increase the feature point offset, thereby obtaining a corrected deformation field by compressing the deformation field.

[0108] In the above embodiment, on the basis of the angle deviation, the deformation field is also corrected in combination with the size deviation between the reference face size and the target face size, which can further ensure the matching degree between the corrected deformation field and the target face, and thus ensure that the updated face image obtained according to the corrected deformation field can retain the characteristics of the target face, which is conducive to improving the face image processing effect.

[0109] In one embodiment, a deformation field of a target face relative to a reference face is determined based on respective mapping positions of facial feature points in each selected two-dimensional plane, including: determining a plurality of facial feature points; for each facial feature point, determining a feature point offset of the target face relative to the reference face according to the respective mapping positions of the facial feature points in each selected two-dimensional plane; and performing offset interpolation on the target face based on the offset of each feature point to obtain a deformation field of the target face relative to the reference face.

[0110] Among them, the specific definition of facial feature points can be found above and will not be repeated here. Specifically, the server can determine multiple facial feature points. Then, the same facial feature point in different three-dimensional face models is mapped to the corresponding selected two-dimensional plane to obtain the corresponding mapping positions of the facial feature points in different three-dimensional face models. For each facial feature point, the server can determine the feature point offset of the target face relative to the reference face based on the respective mapping positions of the facial feature point in each selected two-dimensional plane, thereby obtaining the feature point offset corresponding to each of the multiple facial feature points. In a specific embodiment, as Figure 4 As shown, the hollow point A is the feature point mapping location corresponding to the reference face, and the solid point B is the feature point mapping location corresponding to the target face. It can be understood that through feature point mapping and offset calculation, a sparse offset can be obtained. However, to determine the deformation field of the target face relative to the reference face, it is necessary to calculate the offset of each pixel of the target face. The server can interpolate the offset of the target face based on the offset of each feature point to obtain the offset of consecutive pixels, thereby determining the deformation field of the target face relative to the reference face.

[0111] The specific interpolation algorithm used in offset interpolation is not unique. For example, the server can use GPU (graphics processing unit) rasterization to interpolate dense and fine triangles, rectangles, and other structures, or it can use only coarse, large patches to automatically interpolate the offsets of some points within the patch. In one possible implementation, the server can determine the distance between the current pixel and surrounding pixels with known offsets and use this distance as a metric for weighted interpolation, thereby obtaining a smoother deformation field and a more natural deformation result.

[0112] In the above embodiment, after obtaining sparse feature point offsets through feature point mapping, the target face is offset interpolated based on the offsets of each feature point to obtain the deformation field of the target face relative to the reference face, which can ensure that the face deformation transition is more natural and is conducive to further improving the face image processing effect.

[0113] In practical applications, after obtaining the feature point mapping position, the feature point offset can be calculated by determining the alignment position in each selected two-dimensional plane and aligning the face image in the selected two-dimensional plane based on the alignment position. The alignment position can be a position in the face image area, such as Figure 4 The image alignment position is the eyebrow center O; the alignment position may also be a position in a non-face image area, which is not limited here.

[0114] In one embodiment, the facial image processing method further includes: determining an image alignment position of a target face and a reference face. In this embodiment, determining a feature point offset of the target face relative to the reference face based on the respective mapping positions of the facial feature points in each selected two-dimensional plane includes: performing image alignment on the facial images in each selected two-dimensional plane based on the image alignment position, determining an updated mapping position of each facial feature point in each selected two-dimensional plane; and determining, for each facial feature point, a feature point offset of the target face relative to the reference face based on the updated mapping position corresponding to the facial feature point in each selected two-dimensional plane.

[0115] The image alignment position between the target face and the reference face is located within the facial image area. Furthermore, the image alignment position can be a single-pixel area, such as the center of the eyebrows or the tip of the nose, or a multi-pixel area, such as the philtrum or the bridge of the nose. In one specific embodiment, the server can determine the center of the face as the image alignment position to ensure symmetry of the deformation field.

[0116] Specifically, the server can determine the image alignment position of the target face and the reference face, and based on the image alignment position, perform image alignment on the facial images in each selected two-dimensional plane, and determine the updated mapping position of each facial feature point in each selected two-dimensional plane. Then, for each facial feature point, the server can determine the offset of the feature point of the target face relative to the reference face based on the updated mapping position corresponding to the facial feature point in each selected two-dimensional plane. The specific method of image alignment is not unique. For example, alignment can be achieved through rotation and translation transformation, alignment can also be achieved by fitting, or alignment can be achieved with the image alignment position as the non-offset position.

[0117] In a specific implementation, the server may use the image alignment position as the non-offset position and determine the offset of the feature points of the target face relative to the reference face according to the respective mapping positions of the facial feature points in each selected two-dimensional plane.

[0118] The offset of the non-offset position is zero. Figure 5As shown, after obtaining multiple mapping positions through feature point mapping, the server can align the middle of the eyes of the reference face and the target face in the selected target plane, and then determine the feature point offset of each facial feature point based on the corresponding position information of each facial feature point in the target face and the reference face after alignment.

[0119] In practical applications, the image alignment position may be the position of the facial feature point, or it may not be the position of the facial feature point. It is understandable that when the image alignment position is the position of the facial feature point, the facial feature point is called the target feature point. At this time, the server can determine the feature point offset of the target facial feature point to be zero, and determine the feature point offset of each non-target feature point based on the relative position relationship between the other non-target facial feature points and the target facial feature point in the selected projection plane. In the case that the image alignment position does not contain a facial feature point, the nearest facial feature point can also be selected as the target feature point, and weighted interpolation can be performed based on the distance between the target feature point and the image alignment position to determine the feature point offset corresponding to the target feature point, and then the feature point offset of each non-target feature point can be determined based on the relative position relationship between the other non-target facial feature points and the target facial feature point in the selected projection plane.

[0120] In the above embodiment, the image alignment position of the target face and the reference face is first determined, and after the image alignment is completed based on the image alignment position, the feature point offset of the target face relative to the reference face is determined according to the updated mapping position obtained by the image alignment. This can ensure that the offset of each feature point is determined on the same basis, which is conducive to further improving the face image processing effect.

[0121] In a specific embodiment, determining the image alignment position of a target face and a reference face includes: determining facial feature information of the target face based on a three-dimensional face model of the target face; and determining the associated position of the facial feature information in a selected two-dimensional plane as the image alignment position of the target face and the reference face.

[0122] Facial feature information can be used to describe facial features, such as facial features or contour features. Examples of facial feature information include "widely spaced eyes," "long philtrum," "small eyes," or "large face." Specifically, the server can determine the facial feature information of the target face based on the target face's three-dimensional facial model, and then determine the associated position of the facial feature information in a selected two-dimensional plane as the image alignment position for the target face and the reference face. For example, if the target face has widely spaced eyes, the center of the eyebrows can be determined as the image alignment position; if the target face has a long philtrum, the philtrum position can be used as the image alignment position.

[0123] In the above embodiment, the associated position of the facial feature information in the selected two-dimensional plane is determined as the image alignment position of the target face and the reference face, which can retain the original features of the target face as much as possible and is conducive to further improving the facial image processing effect.

[0124] In one embodiment, Figure 6 As shown, a face image processing method is provided, which can be executed by a computer device. The computer device can be Figure 1 The terminal or server shown is taken as an example. In this embodiment, the method includes the following steps:

[0125] Step S601, obtaining two-dimensional image information of the reference face and the target face;

[0126] Step S602: performing expression recognition on the reference face and the target face based on the two-dimensional image information to determine the expression features of the reference face and the target face;

[0127] Step S603, obtaining a standard expressionless feature corresponding to the standard expressionless image;

[0128] Step S604: for each two-dimensional image information, if the expression feature corresponding to the two-dimensional image information does not meet the similarity condition with the standard expressionless feature, replace the expression of the face represented by the two-dimensional image information based on the standard expressionless image to obtain updated image information of the face;

[0129] Step S605 , based on each updated image information, three-dimensionally reconstructing the reference face and the target face to obtain three-dimensional face models of the reference face and the target face respectively;

[0130] Step S606, determining the facial angle corresponding to the frontal face posture as the selected facial angle;

[0131] Step S607, determining a selected two-dimensional plane represented by the selected facial angle in the three-dimensional space where each three-dimensional face model is located;

[0132] Step S608, determining a plurality of facial feature points and facial feature information of the target face based on the three-dimensional face model of the target face;

[0133] Step S609, determining the associated position of the facial feature information in the selected two-dimensional plane as the image alignment position of the target face and the reference face;

[0134] Step S610, based on the image alignment position, performing image alignment on the face images in each selected two-dimensional plane to determine the updated mapping position of each facial feature point;

[0135] Step S611, for each facial feature point, determining the offset of the feature point of the target face relative to the reference face according to the updated mapping position corresponding to the facial feature point in each selected two-dimensional plane;

[0136] Step S612: performing offset interpolation on the target face based on the offset of each feature point to obtain a deformation field of the target face relative to the reference face;

[0137] Step S613, determining a reference face size of the reference face, a target face size of the target face, and an angle deviation between a facial angle of the target face in the two-dimensional image information and the selected facial angle;

[0138] Step S614: Correcting the deformation field based on the angle deviation and the size deviation between the reference face size and the target face size to obtain a corrected deformation field.

[0139] Step S615 , adjusting the two-dimensional image information of the target face according to the correction deformation field to obtain an updated facial image of the target face.

[0140] In the above-mentioned facial image processing process, the transformation and alignment method from two-dimensional to three-dimensional and then to two-dimensional can obtain the deformation field of the target face relative to the reference face without being affected by the postures of the reference face and the target face, which is conducive to improving the flexibility of facial image processing and improving the facial image processing effect; and, based on the mapping position of the facial feature points, the deformation field is determined so that the obtained deformation field conforms to the facial features of the target face, thereby ensuring that an updated facial image can be obtained according to the deformation field, which can not only fit the template effect of the reference face but also reflect the features of the target face, which is conducive to further improving the effect of facial image processing.

[0141] In the field of image special effects processing, people often use various software and applications to beautify faces and change their appearance. Among them, the face deformation function is a common special effects processing, which can make the face look thinner and more delicate by changing the shape and contour of the face; it can also achieve a cute and magical deformation effect by partially fattening or adjusting the shape of the face. However, traditional technologies have some limitations in deformation. For example, it is impossible to intelligently adjust the face shape according to the distribution characteristics of the facial features to achieve a more natural and template-compliant effect; or it can only use a fixed deformation template to adjust, and users cannot flexibly adjust it themselves. In some technologies, such as Figure 7 As shown, a standard face needs to be given in advance, adjustments need to be made on the standard face, and then the deformation field of the standard face is mapped to the target face. This has high requirements for users and is generally only produced by trained professional special effects generators.

[0142] In the field of image special effects processing, there are already some platforms that support deformation templates such as face thinning and liquefaction. These technologies are usually based on the adjustment of a standard facial deformation field. For example, Figure 8 As shown, after adjusting the target face F, an updated face F' can be obtained with enlarged eyes, narrowed nose, enlarged lips, and narrowed face contour. This will undoubtedly lead to the loss of the original facial features of the target face F, resulting in poor facial image processing effect.

[0143] This application proposes a facial image processing solution that enables adaptive adjustment of facial features. This solution only requires users to upload a reference facial image to support real-time face morphing. Furthermore, the updated face image of the target face maintains the target face's facial features while conforming to the template features of the reference face. This adaptive deformation field strategy based on facial features solves the problems of existing technologies and enables more accurate, intelligent, and natural face morphing.

[0144] In a specific embodiment, Figure 9 As shown, the face image processing method provided in this application may include the following steps:

[0145] Step 1: First, based on face detection, key point recognition and other technologies, obtain the reference face F m2d The 2D facial feature points kp are then used to back-project the 2D feature points back to the pixel space error constraint to perform 3D reconstruction of the reference face. The pose of the reference face in 3D space and the facial morphology fitting result F in 3D space are obtained. M3d , by changing F M3d Project the cluster along the Z axis, that is, F M3d Mapped to the Z plane space, a set of 2D coordinate points KP corresponding to the serial numbers of the 2D feature points are obtained. M :

[0146] pose,F M3d =ReConstruction(F m )

[0147] KP M =Proj(F M3d )

[0148] Step 2: For the real face input Fr (i.e., the target face image) captured by the camera, the Z plane projection point KP of the target face can also be obtained through 3D reconstruction and 2D projection through the processing process of step 1. r , and then after the key point matching alignment mapping operation of the eye, we can get the KP M 2D feature points KP in the same plane space and scale spacer ', that is Figure 9 2D face alignment step in .

[0149] From steps 1 and 2, we can determine that Figure 8 Compared with the traditional face deformation scheme shown in the figure, this application takes into account the differences in the distribution of facial features of actual users by first aligning the actual input face with the given reference face, and can more accurately calculate the deformation field for the actual face input.

[0150] Step 3: After face alignment, you can use the obtained KP r 'With KP M The deformation field Diff_xy is calculated. However, this set of offsets is in the projected scale space of the reference face and belongs to the frontal face pose. It is not at the same angle and scale as the pose position of the real face. In this case, Diff_xy needs to be corrected according to the face angle theta and face size scale of the real face.

[0151] Diff_xy = (KP r '_x- KP M _x , KP r '_y- KP M _y)

[0152] Diff_xy'=Correction(Diff_xy,theta,scale)

[0153] Step 4: After step 3, a set of offsets Diff_xy' corresponding to facial feature points will be obtained, but this set of offsets is sparse. To offset each pixel of the actual input face, the offset of each pixel must be calculated. At this time, the offset of each pixel can be obtained by interpolation and other methods, and then the deformation field can be obtained. Specifically, GPU rasterization can be used for interpolation, through dense and fine triangular faces, rectangles and other structures, or only rough large faces can be set to automatically interpolate the offsets of some points in the face. It can also be used as a metric based on the distance between the current pixel and the surrounding pixels with known offsets to perform weighted interpolation, which can obtain a smoother deformation field and the deformed result will be more natural.

[0154] The above-described facial image processing method can generate a personalized deformation field for a specific facial feature ratio. Compared to standard deformation, the deformation result can both reflect the template effect of the reference face and retain the facial features of the target face, which is conducive to improving the deformation effect and, in turn, the user experience. Furthermore, this application does not require the posture and bilateral symmetry of the two-dimensional image information of the reference face, and can obtain the deformation field corresponding to any selected two-dimensional plane, which is conducive to improving the flexibility of the facial image processing method.

[0155] The present application also provides an application scenario in which the above-mentioned face image processing method is applied.

[0156] In a specific embodiment, the application scenario can be, for example, a live video broadcast scenario. In this application scenario, the terminal can receive the host's operations through an interface. Specifically, the terminal can capture the host's facial image to obtain two-dimensional image information of the target face; and can also obtain a reference facial image selected by the host to obtain two-dimensional image information of the reference face. The reference facial image can be one of multiple reference facial images provided by a server or a reference facial image uploaded by the host through the terminal. Thus, the terminal can obtain two-dimensional image information of each of the reference and target faces. Based on the two-dimensional image information, the terminal performs three-dimensional reconstruction of the reference and target faces to obtain three-dimensional facial models of each. Next, based on the mapping positions of the facial feature points in the selected two-dimensional planes of each three-dimensional facial model, the terminal determines the deformation field of the target face relative to the reference face, where the selected two-dimensional planes of different three-dimensional facial models correspond to the same facial angle. Finally, the two-dimensional image information of the target face is adjusted based on the deformation field to obtain an updated facial image of the target face, which is fed back to the terminal for display.

[0157] Furthermore, in a live broadcast scenario, multiple image frames of the target face are usually collected continuously. In this case, the server can construct a three-dimensional face model based on the collected image frames and obtain the deformation field of the target face relative to the reference face in the frontal posture. Then, the face angle is identified for each image frame respectively, and the deformation field is corrected according to the angular deviation between the facial angle and the facial angle in the frontal posture to obtain the corrected deformation field corresponding to the image frame. Based on the corrected deformation field, the two-dimensional image information of the target face in the image frame is adjusted to obtain the updated image information of the target face in the image frame. Thus, the terminal realizes facial image processing for the video by continuously displaying the facial image updates of each image frame.

[0158] In a specific embodiment, the application scenario can be, for example, a face beautification scenario. In this application scenario, a user can upload a reference face image and a target face image through a terminal, so that the server can obtain the two-dimensional image information of each of the reference face and the target face; based on the two-dimensional image information, the reference face and the target face are respectively reconstructed in three dimensions to obtain three-dimensional face models of the reference face and the target face; then, based on the respective mapping positions of the facial feature points in the selected two-dimensional planes of each three-dimensional face model, the deformation field of the target face relative to the reference face is determined, wherein the selected two-dimensional planes of different three-dimensional face models correspond to the same facial angle; finally, the two-dimensional image information of the target face is adjusted according to the deformation field, and an updated face image of the target face is obtained and fed back to the terminal, so that the terminal can display an updated face image that not only fits the template effect of the reference face but also reflects the characteristics of the host's own face.

[0159] By adopting the above-mentioned facial image processing method and an adaptive deformation adjustment strategy based on facial feature points, an updated facial image can be obtained that not only fits the template effect of the reference face but also reflects the characteristics of the host's own face. This provides a more accurate, intelligent and natural facial deformation function, which is conducive to improving the facial image processing effect.

[0160] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0161] Based on the same inventive concept, embodiments of the present application also provide a facial image processing device for implementing the aforementioned facial image processing method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations in one or more facial image processing device embodiments provided below can be found in the above-mentioned limitations on the facial image processing method and will not be further elaborated here.

[0162] In one embodiment, Figure 10As shown, a face image processing device is provided, comprising: an image information acquisition module 1001, a three-dimensional reconstruction module 1002, a deformation field determination module 1003 and an image update module 1004, wherein:

[0163] Image information acquisition module 1001, used to acquire two-dimensional image information of the reference face and the target face;

[0164] A 3D reconstruction module 1002 is configured to perform 3D reconstruction on the reference face and the target face based on the 2D image information to obtain 3D face models of the reference face and the target face respectively.

[0165] A deformation field determination module 1003 is configured to determine the deformation field of the target face relative to the reference face based on the respective mapping positions of the facial feature points in the selected two-dimensional planes of each three-dimensional face model; the selected two-dimensional planes of different three-dimensional face models correspond to the same facial angle;

[0166] The image updating module 1004 is configured to adjust the two-dimensional image information of the target face according to the deformation field to obtain an updated facial image of the target face.

[0167] In one embodiment, the three-dimensional reconstruction module 1002 includes: an expression recognition unit, which is used to perform expression recognition on the reference face and the target face respectively based on each two-dimensional image information, and determine the expression features of the reference face and the target face respectively; an expression replacement unit, which is used to replace the expression of at least one of the reference face and the target face when the similarity conditions are not met between the expression features, and obtain updated image information; a three-dimensional reconstruction unit, which is used to perform three-dimensional reconstruction on the reference face and the target face respectively based on each updated image information, and obtain three-dimensional face models of the reference face and the target face respectively.

[0168] In one embodiment, the expression replacement unit is specifically used to: obtain standard expression features corresponding to the standard expression image; for each two-dimensional image information, when the expression features corresponding to the two-dimensional image information and the standard expression features do not meet the similarity conditions, replace the expression of the face represented by the two-dimensional image information based on the standard expression image to obtain updated image information of the face.

[0169] In one embodiment, the deformation field determination module 1003 includes: a selected two-dimensional plane determination unit, used to determine the selected two-dimensional plane represented by the selected facial angle in the three-dimensional space where each three-dimensional face model is located; a deformation field determination unit, used to determine the deformation field of the target face relative to the reference face based on the respective mapping positions of the facial feature points in each selected two-dimensional plane.

[0170] In one embodiment, the facial image processing device also includes: a base facial angle selection module, which is used to perform facial angle recognition on the target face based on the two-dimensional image information of the target face, determine the facial angle of the target face; and determine the facial angle of the target face as the selected facial angle.

[0171] In one embodiment, the facial image processing apparatus further includes a base facial angle selection module configured to determine the facial angle corresponding to the frontal facial posture as the selected facial angle. In this embodiment, the image update module 1004 includes an angle deviation determination unit configured to determine the angle deviation between the facial angle of the target face in the two-dimensional image information and the selected facial angle; a deformation field correction unit configured to correct the deformation field based on the angle deviation to obtain a corrected deformation field; and an image update unit configured to adjust the two-dimensional image information of the target face based on the corrected deformation field to obtain an updated facial image of the target face.

[0172] In one embodiment, the facial image processing apparatus further includes a facial size determination module configured to determine a reference facial size of a reference face and a target facial size of a target face. In this embodiment, the deformation field correction unit is configured to correct the deformation field based on the angular deviation and the size deviation between the reference and target facial sizes to obtain a corrected deformation field.

[0173] In one embodiment, the deformation field determination unit includes: a facial feature point determination component for determining multiple facial feature points; a mapping component for determining, for each facial feature point, the feature point offset of the target face relative to the reference face based on the respective mapping positions of the facial feature points in each selected two-dimensional plane; and an interpolation component for performing offset interpolation on the target face based on the offset of each feature point to obtain the deformation field of the target face relative to the reference face.

[0174] In one embodiment, the facial image processing apparatus further includes an alignment position determination module configured to determine the image alignment position of a target face and a reference face. In this embodiment, the mapping component is configured to: perform image alignment on the facial images in each selected two-dimensional plane based on the image alignment position, determine the updated mapping position of each facial feature point in each selected two-dimensional plane, and determine, for each facial feature point, an offset of the target face feature point relative to the reference face feature point based on the updated mapping position corresponding to the facial feature point in each selected two-dimensional plane.

[0175] In a specific embodiment, determining the image alignment position of a target face and a reference face includes: determining facial feature information of the target face based on a three-dimensional face model of the target face; and determining the associated position of the facial feature information in a selected two-dimensional plane as the image alignment position of the target face and the reference face.

[0176] In one specific embodiment, the two-dimensional image information of the target face includes multiple two-dimensional image frames. In this embodiment, the three-dimensional reconstruction module 1002 is specifically configured to: perform facial pose recognition on each two-dimensional image frame to determine the facial pose of each two-dimensional image frame; determine the depth information of the facial key points based on the facial pose and the position information of the same facial key points in each two-dimensional image frame; and perform three-dimensional reconstruction of the target face based on the position information and depth information of the multiple facial key points to obtain a three-dimensional facial model of the target face.

[0177] Each module in the facial image processing device described above may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0178] In one embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 11 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data information involved in the above-mentioned facial image processing process. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a facial image processing method is implemented.

[0179] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 12As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means. The wireless means can be implemented via Wi-Fi, a mobile cellular network, NFC (near field communication), or other technologies. When executed by the processor, the computer program implements a facial image processing method. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0180] Those skilled in the art will understand that Figure 11 or Figure 12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0181] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0182] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0183] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0184] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant regions and areas. Furthermore, the user may choose not to authorize the use of such information and related data, or may refuse or conveniently refuse to receive push notifications.

[0185] In this application, when collecting and processing relevant data during the instance application, the requirements of relevant local laws and regulations should be strictly followed to obtain the informed consent or separate consent of the personal information subject, and subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0186] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0187] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A facial image processing method, characterized in that: The method comprises: Obtaining two-dimensional image information of the reference face and the target face; Based on the two-dimensional image information, the reference face and the target face are respectively reconstructed in three dimensions to obtain three-dimensional face models of the reference face and the target face; determining a deformation field of the target face relative to the reference face based on respective mapping positions of facial feature points in the selected two-dimensional planes of each of the three-dimensional face models; the selected two-dimensional planes of different three-dimensional face models correspond to the same facial angle; The two-dimensional image information of the target face is adjusted according to the deformation field to obtain an updated facial image of the target face.

2. The method according to claim 1, characterized in that The step of reconstructing the reference face and the target face in three dimensions based on the two-dimensional image information to obtain three-dimensional face models of the reference face and the target face, respectively, includes: Based on each of the two-dimensional image information, expression recognition is performed on the reference face and the target face to determine the expression features of the reference face and the target face respectively; When the expression features do not satisfy a similarity condition, performing expression replacement on at least one of the reference face and the target face to obtain updated image information; Based on each of the updated image information, the reference face and the target face are respectively reconstructed in three dimensions to obtain respective three-dimensional face models of the reference face and the target face.

3. The method according to claim 2, characterized in that When the expression features do not satisfy the similarity condition, performing expression replacement on at least one of the reference face and the target face to obtain updated image information includes: Obtaining standard expression features corresponding to standard expression images; For each of the two-dimensional image information, when the expression features corresponding to the two-dimensional image information do not meet the similarity conditions with the standard expression features, the expression of the face represented by the two-dimensional image information is replaced based on the standard expression image to obtain updated image information of the face.

4. The method according to claim 1, wherein Determining the deformation field of the target face relative to the reference face based on respective mapping positions of facial feature points in the selected two-dimensional plane of each of the three-dimensional face models includes: Determining a selected two-dimensional plane represented by a selected facial angle in the three-dimensional space where each of the three-dimensional face models is located; Based on the respective mapping positions of the facial feature points in the selected two-dimensional planes, a deformation field of the target face relative to the reference face is determined.

5. The method according to claim 4, characterized in that The method further comprises: Based on the two-dimensional image information of the target face, performing facial angle recognition on the target face to determine the facial angle of the target face; The facial angle of the target face is determined as a selected facial angle.

6. The method according to claim 4, characterized in that The method further comprises: determining the facial angle corresponding to the frontal face posture as the selected facial angle; The adjusting the two-dimensional image information of the target face according to the deformation field to obtain an updated facial image of the target face includes: determining an angular deviation between a facial angle of the target face in the two-dimensional image information and the selected facial angle; Correcting the deformation field according to the angular deviation to obtain a corrected deformation field; The two-dimensional image information of the target face is adjusted according to the correction deformation field to obtain an updated facial image of the target face.

7. The method according to claim 6, characterized in that The method further comprises: determining a reference face size of the reference face and a target face size of the target face; Correcting the deformation field according to the angle deviation to obtain a corrected deformation field includes: The deformation field is corrected according to the angle deviation and the size deviation between the reference face size and the target face size to obtain a corrected deformation field.

8. The method according to claim 4, characterized in that The determining of the deformation field of the target face relative to the reference face based on the respective mapping positions of the facial feature points in the selected two-dimensional planes includes: Determine multiple facial feature points; For each of the facial feature points, determining an offset of the feature point of the target face relative to the reference face according to a respective mapping position of the facial feature point in each of the selected two-dimensional planes; The target face is subjected to offset interpolation based on the offsets of the feature points to obtain a deformation field of the target face relative to the reference face.

9. The method according to claim 8, characterized in that The method further comprises: Determining an image alignment position of the target face and the reference face; Determining the offset of the feature points of the target face relative to the reference face according to the respective mapping positions of the facial feature points in the selected two-dimensional planes includes: Based on the image alignment position, performing image alignment on the facial images in each of the selected two-dimensional planes to determine an updated mapping position of each of the facial feature points; For each of the facial feature points, an offset of the feature point of the target face relative to the reference face is determined according to the updated mapping position corresponding to the facial feature point in each of the selected two-dimensional planes.

10. The method according to claim 9, characterized in that Determining the image alignment position of the target face and the reference face includes: Determining facial feature information of the target face based on the three-dimensional face model of the target face; The associated position of the facial feature information in the selected two-dimensional plane is determined as the image alignment position of the target face and the reference face.

11. The method according to any one of claims 1 to 10, characterized in that The two-dimensional image information of the target face includes a plurality of two-dimensional image frames; and the process of obtaining a three-dimensional face model of the target face includes: Performing facial posture recognition on each of the two-dimensional image frames to determine the facial posture of each of the two-dimensional image frames; Determining depth information of the facial key points based on the facial postures and respective position information of the same facial key points in the two-dimensional image frames; The target face is three-dimensionally reconstructed based on the position information and depth information of each of the multiple facial key points to obtain a three-dimensional face model of the target face.

12. A facial image processing device, characterized in that: The device comprises: An image information acquisition module is used to acquire two-dimensional image information of the reference face and the target face; a three-dimensional reconstruction module, configured to perform three-dimensional reconstruction on the reference face and the target face based on the two-dimensional image information, to obtain three-dimensional face models of the reference face and the target face respectively; a deformation field determination module, configured to determine the deformation field of the target face relative to the reference face based on the respective mapping positions of the facial feature points in the selected two-dimensional planes of each of the three-dimensional face models; the selected two-dimensional planes of different three-dimensional face models correspond to the same facial angle; An image updating module is used to adjust the two-dimensional image information of the target face according to the deformation field to obtain an updated facial image of the target face.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images