Facial driving method and apparatus

Abstract

Embodiments of the present application provide a face driving method, which comprises: acquiring a first facial feature of a to-be-driven image; acquiring a second facial feature and a key point feature of a driving source image; acquiring a depth feature according to the key point feature and the second facial feature; inputting the first facial feature, the second facial feature, the key point feature and the depth feature into a pre-trained target neural network, and outputting a target driving image through the target neural network; wherein the target driving image is an image formed by the to-be-driven image under the driving of the driving source image. The technical scheme of the embodiments of the present application can accurately analyze the three-dimensional structure of the face of the driving source image by using the depth feature, thereby providing more abundant face information for face driving, and effectively improving the performance and precision of face driving.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, and in particular to a face driving method and device, computer equipment and a computer readable storage medium. BACKGROUND

[0002] With the rapid development of neural networks and computer technology, face driving technology has been applied to the fields of computer games, animation production, etc. The face driving technology can drive a three-dimensional renderer face model or a virtual face of a 2D image based on a real face, so as to realize the synchronization of the virtual face and the real face in terms of speaking, expression and facial details, etc.

[0003] The existing face driving technology has poor driving effect, thereby affecting the user experience.

[0004] It should be noted that the above content is not necessarily prior art, and is not used to limit the patent protection scope of the present application. SUMMARY

[0005] Embodiments of the present application provide a face driving method, device, computer equipment and computer readable storage medium to solve or alleviate one or more technical problems proposed above.

[0006] One aspect of an embodiment of the present application provides a face driving method, the method comprising:

[0007] obtaining a first facial feature of a to-be-driven image;

[0008] obtaining a second facial feature and a key point feature of a driving source image;

[0009] obtaining a depth feature according to the key point feature and the second facial feature;

[0010] inputting the first facial feature, the second facial feature, the key point feature and the depth feature into a pre-trained target neural network, and outputting a target driving image by the target neural network; wherein the target driving image is an image formed by the to-be-driven image under the driving of the driving source image.

[0011] Optionally, the obtaining of the second facial feature and the key point feature of the driving source image comprises:

[0012] obtaining 3D point cloud information of a key point in the driving source image by a pre-trained 3D model;

[0013] The 3D point cloud information includes a translation parameter and a rotation parameter; the translation parameter is used to represent a moving distance of the key point in a three-dimensional coordinate system; and the rotation parameter is used to represent a rotation angle of the key point in the three-dimensional coordinate system.

[0014] Optionally, the obtaining of the depth feature according to the key point feature map and the second face feature includes:

[0015] The first fusion feature is obtained according to the key point feature and the second face feature.

[0016] The first fusion feature is input into a pre-trained face depth network to obtain the depth feature.

[0017] Optionally, the obtaining of the first fusion feature according to the key point feature and the second face feature includes:

[0018] The key point feature is loaded into the second face feature by modulation and demodulation to obtain the first fusion feature.

[0019] Optionally, the target neural network includes a cross-attention mechanism network.

[0020] Optionally, the cross-attention mechanism network includes an encoding layer, a first convolutional layer, a second convolutional layer and a third convolutional layer; and correspondingly, the outputting of the target driving image by the target neural network includes:

[0021] The second fusion feature is obtained according to the key point feature and the depth feature.

[0022] The second fusion feature is input into the encoding layer.

[0023] The output data of the encoding layer is subjected to a convolution operation via the first convolutional layer to obtain a first matrix.

[0024] The third fusion feature is obtained according to the first face feature and the second face feature.

[0025] The third fusion feature is input into the second convolutional layer to obtain a second matrix.

[0026] The third fusion feature is input into the third convolutional layer to obtain a third matrix.

[0027] The optical flow feature is obtained according to the first matrix and the second matrix.

[0028] The target driving image is obtained according to the optical flow feature and the third matrix.

[0029] Optionally, the target driving image is acquired according to the optical flow feature and the third matrix, including:

[0030] A dimension value is determined according to the first matrix and the second matrix.

[0031] The target driving image is acquired according to the optical flow feature, the third matrix and the dimension value.

[0032] Another aspect of the embodiments of the present application provides a face driving device, the device comprising:

[0033] A first acquisition module is configured to acquire a first facial feature of a to-be-driven image.

[0034] A second acquisition module is configured to acquire a second facial feature and a key point feature of a driving source image, wherein the driving source image is used to drive the to-be-driven image to form a target driving image.

[0035] A third acquisition module is configured to acquire a depth feature according to the key point feature and the second facial feature.

[0036] An output module is configured to input the first facial feature, the second facial feature, the key point feature and the depth feature into a pre-trained target neural network, and output the target driving image through the target neural network.

[0037] Another aspect of the embodiments of the present application provides a computer device, comprising:

[0038] At least one processor; and

[0039] A memory in communication connection with the at least one processor;

[0040] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0041] Another aspect of the embodiments of the present application provides a computer readable storage medium, and the computer readable storage medium stores computer instructions, and the computer instructions are executed by a processor to implement the method as described above.

[0042] The embodiments of the present application can include the following advantages by adopting the above technical solutions:

[0043] The second facial feature and the key point feature of the driving source image are used to obtain the depth feature of the driving source image. Then, during the process of driving the to-be-driven image by the driving source image, the first facial feature of the to-be-driven image can be influenced by combining the key point feature, the depth feature and the second facial feature of the driving source image, so as to realize accurate synchronization of actions, expressions and the like. Since the depth feature is used to accurately analyze the three-dimensional structure of the face of the driving source image, more abundant face information is provided for face driving, and the performance effect and the accuracy of face driving are effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain exemplary implementations of the application. The illustrated embodiments are merely examples and do not limit the scope of the claims. In all the drawings, like reference numerals refer to like parts throughout the several views.

[0045] Figure 1 An operation environment diagram of the face driving method according to Embodiment One of the present application is schematically shown;

[0046] Figure 2 A flowchart of the face driving method according to Embodiment One of the present application is schematically shown;

[0047] Figure 3 A flowchart of the sub-steps in step S202 is schematically shown; Figure 2

[0048] A flowchart of the sub-steps in step S204 is schematically shown; Figure 4 Figure 2 A flowchart of the sub-steps in step S206 is schematically shown;

[0049] Figure 5 Figure 4 A flowchart of the sub-steps in step S400 is schematically shown;

[0050] Figure 6 A flowchart of the sub-steps in step S402 is schematically shown; Figure 2

[0051] A flowchart of the sub-steps in step S614 is schematically shown; Figure 7 Figure 6 An application example diagram of the face driving method according to Embodiment One of the present application is schematically shown;

[0052] Figures 8A to 8C A block diagram of the face driving device according to Embodiment Two of the present application is schematically shown; and

[0053] Figure 9

[0054] Figure 10 ​​​​The hardware architecture schematic diagram of the computer device according to the embodiment three of the present application is schematically shown. DETAILED DESCRIPTION

[0055] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0056] It should be noted that the description of "first", "second" and the like in the embodiments of the present application is only for the purpose of description and cannot be understood as indicating or implying the relative importance of the technical features indicated or implying the number of technical features indicated. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the fact that those of ordinary skill in the art can realize it. When the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist and is not within the scope of protection claimed by the present application.

[0057] In the description of the present application, it should be understood that the numerical reference number before the step does not indicate the order of execution of the steps before and after the step, but is only used to facilitate the description of the present application and to distinguish each step, therefore it cannot be understood as a limitation of the present application.

[0058] Firstly, the explanation of the terms related to the present application is provided:

[0059] Face depth: refers to the three-dimensional depth of a face, including concave and convex parts, face shape, expression, etc. By using a camera or sensor with depth perception technology, a face depth map can be obtained. The face depth map can accurately show the depth information of the face surface. In the field of computer vision, the face depth map is used in combination with a 2D face image for face recognition, expression analysis, virtual face changing, etc.

[0060] Face driving: driving the face of a three-dimensional renderer face model or a 2D image by a real face, so as to realize the synchronization of virtual face and real face in terms of speaking, expression and facial details, etc.

[0061] Face key points: refers to some special positions on a face, such as eyes, nose, mouth, etc. Face key points can be used to represent the posture, expression, etc. of a face. Face key points can be automatically detected and marked by a face key point detection algorithm, providing basic support for subsequent face recognition, expression recognition, expression synthesis, etc.

[0062] Self-supervised learning: a learning method in machine learning. Self-supervised learning does not require manually labeled label data, but uses unlabeled data to generate corresponding training data, and then trains through these automatically generated "pseudo labels" to realize model learning and prediction. Self-supervised learning is widely used in computer vision, natural language processing and speech recognition.

[0063] Attention mechanism: a technique in artificial neural networks that simulates cognitive attention. This mechanism can enhance the weight of some parts of the input data of the neural network, while reducing the weight of other parts, to help the neural network better focus on important parts of the input data, thereby improving the performance and interpretability of the model. Attention mechanism is widely used in deep learning, including natural language processing, computer vision, speech recognition and other fields.

[0064] Optical flow: refers to the movement of target pixels in an image due to the movement of objects in the image or the movement of the camera between two consecutive images.

[0065] Point cloud: a digital 3D representation of a physical object composed of multiple measurement points.

[0066] Secondly, in order to facilitate the understanding of the technical solutions provided by the person skilled in the art, the related technology is described as follows:

[0067] Face-driven technology can be used to drive facial expressions and gestures. That is, through the principle of muscle movement, how facial expressions reflect changes in facial muscles is studied. There are several solutions for face recognition technology: (1) based on pre-set rules, facial expressions and gestures of virtual faces are generated through face modeling and computer simulation. This face-driven method needs to set rules artificially, and it is difficult to adapt to the complexity and variability of human expressions. (2) through neural network technology (deep learning) to extract features and recognize expressions of faces, facial expressions and gestures can be captured, and more intuitive and detailed expression synthesis can be achieved. It should be noted that face-driven technology based on neural network technology can be applied in virtual reality, computer games, animation production and other fields.

[0068] The applicant understands that related face-driven technology is to first obtain 2D images or videos of real faces, and then extract facial key points of real faces from images through face detection algorithm. The facial key points of real faces are input into a simple neural network to predict and drive virtual faces.

[0069] However, the aforementioned face-driven technologies have several drawbacks: (1) Face-driven technologies based on simple neural networks require face detection first, and the accuracy of face detection directly affects the face-driven effect. (2) Incomplete facial feature extraction leads to unsatisfactory face-driven effects. (3) The lack of facial depth information in facial key points also reduces the effect and accuracy of face-driven technologies. (4) 2D-based face-driven technologies can only provide facial expression information from a two-dimensional perspective. In richer virtual scenes, more realistic facial expression effects are needed to meet the requirements.

[0070] Therefore, this application provides a face-driven technical solution. In this technical solution, (1) a 3D model is used to extract facial information and a self-supervised face depth map network is trained to provide richer facial information. (2) A cross-attention mechanism is used to better explore the correlation between facial and 3D information, which can achieve more refined facial expression capture and reflect more facial details. (3) This solution can use any face image instead of a video as before, and can synchronize facial expressions and speech with the face in the video source. It has the advantages of simplicity and strong versatility. It can be seen that a high-precision face depth map is trained by self-supervised training based on the 3D information of the face to provide richer facial information, not just simple information of key points. Secondly, this solution uses a cross-attention mechanism to better extract and utilize information to achieve a more accurate face-driven effect. Therefore, this solution uses 3D information to make the capture of facial expressions more refined and reflect more facial details; during expression switching, more natural facial expression dynamics can be achieved, thereby achieving a more realistic facial expression synthesis effect. See the following text for details.

[0071] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0072] like Figure 1 As shown, the runtime environment diagram includes: service platform 2, clients (4A, 4B, ..., 4N).

[0073] Service platform 2 can connect to clients (4A, 4B, ..., 4N) via the network.

[0074] Service platform 2 can be a single server, a server cluster, or a cloud computing service center.

[0075] Service platform 2 can provide face detection services, face-driven services, etc. to clients.

[0076] Face detection services can include facial feature extraction services, facial landmark detection services, etc.

[0077] The facial feature extraction service can be configured to receive images and to extract facial features from those images.

[0078] The facial landmark detection service can be configured to receive images and to automatically detect and label facial landmarks in the images.

[0079] The face-driven service can be configured to receive an image to be driven and a source image, and to drive the image to be driven based on the source image.

[0080] The face-driven service can also be configured to receive driver instructions and to perform face driving based on the driver instructions.

[0081] Service platform 2 can be located in a data center, such as a single location, or distributed across different geographical locations (e.g., multiple locations). Service platform 2 can provide services via a network. The network includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or similar devices. The network can include physical links, such as coaxial cable links, twisted-pair cable links, fiber optic links, or combinations thereof, or wireless links, such as cellular links, satellite links, Wi-Fi links, etc.

[0082] Clients (4A, 4B, ..., 4N) can be configured to access the content and services of service platform 2. Clients (4A, 4B, ..., 4N) can include electronic devices with built-in or external display panels, such as mobile devices, tablets, laptops, workstations, virtual reality devices, gaming devices, digital streaming media devices, vehicle terminals, smart TVs, set-top boxes, etc., and can also include virtualized computing instances. Virtualized computing instances can include virtual machines, such as simulations of computer systems, operating systems, servers, etc. The computing device can load the virtual machine based on the virtual image and / or other data defining specific software (e.g., operating system, dedicated applications, servers) used for simulation. As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computing devices.

[0083] A client (4A, 4B, ..., 4N) can be associated with one or more users. A single user can also use one or more of the clients (4A, 4B, ..., 4N) to access service platform 2. Clients (4A, 4B, ..., 4N) can travel to various locations and use different networks to access service platform 2.

[0084] The client (4A, 4B, ..., 4N) may include an interface. This interface may include a touchpad, touchscreen, mouse, keyboard, or other sensing elements. For example, the input element may be configured to receive user commands that cause the client (4A, 4B, ..., 4N) to perform various operations, such as selecting an image to be driven, selecting a driving source image, uploading an image to be driven, uploading a driving source image or video, confirming face driving, canceling face driving, etc. In some embodiments, the client may also perform image driving locally.

[0085] It should be noted that the above-mentioned equipment is exemplary, and the number and type of equipment can be adjusted in different scenarios or according to different needs.

[0086] The technical solution of this application will be described below through multiple embodiments, using service platform 2 as the implementing entity. It should be understood that these embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments described herein.

[0087] Example 1

[0088] Figure 2 A flowchart illustrating a face-driven method according to Embodiment 1 of this application is shown schematically.

[0089] like Figure 2 As shown, the face-driving method may include steps S200 to S206, wherein:

[0090] Step S200: Obtain the first facial features of the image to be driven.

[0091] Step S202: Obtain the second facial features and key point features of the driving source image.

[0092] Step S204: Obtain depth features based on the key point features and the second facial features.

[0093] Step S206: Input the first facial feature, the second facial feature, the key point feature, and the depth feature into a pre-trained target neural network, and output the target driving image through the target neural network; wherein, the target driving image is the image formed by the image to be driven under the driving source image.

[0094] The depth features of the driving source image are obtained by using the second facial features and key point features of the driving source image. Then, during the process of driving the image to be driven using the driving source image, the key point features, depth features, and second facial features of the driving source image can be combined to influence the first facial features of the image to be driven, thereby achieving precise synchronization of actions, expressions, etc. Because the depth features are used to accurately analyze the three-dimensional facial structure of the driving source image, richer facial information is provided for face driving, effectively improving the performance and accuracy of face driving.

[0095] The following combination Figure 2 The steps in steps S200 to S206, as well as other optional steps, are described in detail.

[0096] Step S200 The first facial features of the image to be driven are obtained.

[0097] The image to be driven contains a face to be driven. The face to be driven can be a real face, a virtual face synthesized using deep synthesis technology, or a 3D face model formed by modeling a face model.

[0098] Facial features can include information on specific facial features and overall facial features, such as facial features, facial structure, facial shape and gender, and facial contour. The first facial feature is the facial feature of the image to be driven.

[0099] In this embodiment, the first facial features of the image to be driven can be obtained in various ways, thereby providing basic support for face driving. For example, the first facial features of the image to be driven can be obtained through manual detection and labeling, or the first facial features can be obtained by feature extraction of the image to be driven using neural network technology.

[0100] In an exemplary application, a pre-trained first facial feature extraction network can be used to obtain the first facial features of the image to be driven. The first facial feature extraction network is a pre-trained model used to extract facial features from the image to be driven. In this example, using the first facial feature extraction network to extract features from the image to be driven can effectively improve the accuracy and completeness of facial feature extraction, thereby mitigating the impact of incomplete or inaccurate facial feature extraction on subsequent face-driven performance.

[0101] In some embodiments, the first facial feature extraction network can be a Unet (Convolutional Networks for Biomedical Image Segmentation) model, used to compress data and extract facial features, thereby reducing the amount of data and extracting facial features. It should be noted that the first facial feature extraction network can also be a convolutional neural network (CNN), and different models can be selected according to actual needs; no limitation is made here.

[0102] Step S202 The second facial features and key point features of the driving source image are obtained.

[0103] The driving source image can be a video source, a frame sequence captured by a camera, etc. The driving source image can be used to drive the image to be driven, such as driving the facial expressions and facial details of the image to be driven. The second facial feature is the facial feature of the driving source image.

[0104] Key points can be specific locations on a person's face, such as the eyes, nose, and mouth, or other pre-defined locations.

[0105] In this embodiment, the second facial features and key point features of the driving source image can be obtained in a variety of ways, thereby extracting more complete driving source face information.

[0106] For example, the second facial features of the image to be driven can be obtained through a pre-trained second facial feature extraction network.

[0107] It should be noted that the second facial feature extraction network and the first facial feature extraction network are two independent network models that do not affect each other. The second facial feature extraction network is a pre-trained model used to extract facial features from the driving source image.

[0108] In this embodiment, by using a second facial feature extraction network to extract features from the driving source image, the accuracy of facial feature extraction from the driving source can be effectively improved, and the facial features of the driving source image can be extracted more completely, effectively alleviating the poor performance of subsequent face driving caused by incomplete or inaccurate facial feature extraction from the driving source.

[0109] In some embodiments, the second facial feature extraction network can be a U-net model, used for data compression and facial feature extraction, thereby reducing the amount of data and extracting facial features. It should be noted that the second facial feature extraction network can also be a CNN model, and different models can be selected according to actual needs; no limitation is made here.

[0110] Below is an example scheme for obtaining key point features of a driving source image.

[0111] In optional embodiments, such as Figure 3 As shown, step S202 may include:

[0112] Step S300: Obtain 3D point cloud information of key points in the driving source image using a pre-trained 3D model; wherein, the 3D point cloud information includes translation parameters and rotation parameters; the translation parameters are used to represent the movement distance of the key points in the three-dimensional coordinate system; the rotation parameters are used to represent the rotation angle of the key points in the three-dimensional coordinate system.

[0113] For example, a DECA (Detailed Expression Capture and Animation) model can be used to acquire 3D information from the source image to obtain a 3D point cloud of a face. A point cloud is a digital 3D representation of a physical object, composed of multiple measurement points. Each key point on a face (such as the nose or mouth) is composed of multiple measurement points at the corresponding location in the 3D point cloud. Each measurement point can include six parameters (x, y, z, p, y, r), including three translation parameters (x, y, z) and three rotation parameters (p, y, r). The translation parameters (x, y, z) represent the distance the measurement point moves along the x, y, and z axes in the 3D coordinate system. The rotation parameters represent the rotation angle or Euler angle of the measurement point in the 3D coordinate system, including pitch, yaw, and roll, usually expressed in radians or angles. In the 3D coordinate system, a key point is a collection of multiple measurement points at a corresponding location; therefore, the position and orientation of the key point in 3D space can be described using the parameter information of the corresponding measurement points. It should be noted that other methods can also be used to obtain 3D point cloud information, such as 3D Morphable Model (3DMM, three-dimensional variable model).

[0114] In this embodiment, 3D point cloud information of key points in the driving source image can be obtained through a 3D model. The orientation of the key points in three-dimensional space can be determined based on the translation and rotation parameters in the 3D point cloud information, thereby obtaining face information from a three-dimensional perspective more accurately and providing richer information for subsequent face driving.

[0115] Step S204 Depth features are obtained based on the key point features and the second facial features.

[0116] Key points are specific locations on the face, so key point features can be used to represent information such as the pose and expression of the driving source face. Secondary facial features can include overall and specific feature information of the driving source face, such as facial contours and facial structure. Therefore, in this embodiment, depth features of the driving source image can be extracted based on key point features and secondary facial features, combined with local and overall features, and three-dimensional and two-dimensional perspectives. Depth features are the depth information of the face; they can distinguish facial contours and features, differentiate different depths and curvatures, thereby accurately resolving the three-dimensional structure of the face and providing richer facial information for face driving.

[0117] The following provides an exemplary scheme for obtaining depth features.

[0118] In optional embodiments, such as Figure 4 As shown, step S204 may include:

[0119] Step S400: Obtain the first fusion feature based on the key point features and the second facial features.

[0120] Step S402: Input the first fused feature into the pre-trained facial depth network to obtain the depth feature.

[0121] In this embodiment, before inputting the features into the facial deep network, the key point features and the second facial features can be fused to combine the two features into a more discriminative feature. This feature is then input into the facial neural network, which can accurately parse the 3D structure of the face and extract depth features with lower computational resources. Feature fusion methods can include feature concatenation, feature combination, etc., and can be selected according to actual needs; no limitation is made here.

[0122] In some embodiments, the facial deep network may include a model pre-trained using self-supervised learning. High-precision depth features can be obtained through a self-supervised trained facial deep network. It should be noted that the facial neural network can be a convolutional neural network model or other models, and is not limited thereto.

[0123] The following provides an exemplary scheme for obtaining the first fusion feature.

[0124] In optional embodiments, such as Figure 5 As shown, step S400 may include:

[0125] Step S500: Based on the key point features and the second facial features, the key point features are loaded into the second facial features through modulation and demodulation to obtain the first fused features.

[0126] For example, the keypoint features may include a 512*512 keypoint feature map. The second facial feature may also include a 512*512 facial feature map. The modulation and demodulation method involves adding each feature point value from the keypoint feature map to the corresponding feature point on the facial feature map to obtain a 512*512 fused feature map. This fused feature map corresponds to the first fused feature. It should be noted that the feature point values ​​can be preprocessed before fusion, such as variance calculation.

[0127] In this embodiment, feature fusion is performed using modulation and demodulation, allowing feature points at corresponding locations in feature maps of different modalities to interact and obtain a first fused feature with more facial detail and 3D information. Inputting this first fused feature into a facial depth network effectively extracts facial depth information, thereby more accurately resolving the 3D structure of the face.

[0128] Step S206 The first facial feature, the second facial feature, the key point feature, and the depth feature are input into a pre-trained target neural network, and the target-driven image is output through the target neural network.

[0129] A target neural network can be a network model that allows interaction and information transfer between feature maps.

[0130] In this embodiment, the first facial feature, the second facial feature, key point features, and depth features can be jointly input into the target neural network. The target neural network can interact and transmit information between different feature maps, better explore the correlation between face and 3D information, make facial expression capture more refined, and output the target driving map can reflect more facial details, achieving a more accurate face driving effect.

[0131] In an optional embodiment, the target neural network can be a cross-attention mechanism network. Cross-attention mechanisms help the neural network better focus on important parts of the input data, cross-aligning and comparing feature maps in different semantic spaces and different modalities, optimizing the fusion and alignment of multimodal information, thereby improving the model's performance and understanding ability. It should be noted that the target neural network can also be other models.

[0132] Below is an example scheme for obtaining target-driven images using a cross-attention mechanism network.

[0133] In optional embodiments, the cross-attention mechanism network may include an encoding layer, a first convolutional layer, a second convolutional layer, and a third convolutional layer. Correspondingly, as...Figure 6 As shown, step S206 includes:

[0134] Step S600: Obtain the second fusion feature based on the key point features and the depth features.

[0135] Step S602: Input the second fused feature into the coding layer.

[0136] Step S604: Perform a convolution operation on the output data of the encoding layer via the first convolutional layer to obtain a first matrix.

[0137] Step S606: Obtain the third fusion feature based on the first facial feature and the second facial feature.

[0138] Step S608: Input the third fusion feature into the second convolutional layer to obtain the second matrix.

[0139] Step S610: Input the third fusion feature into the third convolutional layer to obtain the third matrix.

[0140] Step S612: Obtain optical flow features based on the first matrix and the second matrix.

[0141] Step S614: Obtain the target driving image based on the optical flow features and the third matrix.

[0142] In this embodiment, feature fusion can be performed based on keypoint features and depth features to obtain a second fused feature, thereby obtaining more comprehensive and accurate features. Feature fusion can include feature stitching, feature combination, etc. For example, the keypoint feature can be a three-channel keypoint feature map with a size of 512*512, and the depth feature can also be a three-channel depth feature map with a size of 512*512. Since each feature map has the same size, the keypoint feature and the depth feature can be stitched together to obtain a six-channel fused feature map with a size of 512*512, i.e., the second fused feature. Similarly, a 512*512 three-channel first facial feature map and a 512*512 three-channel second facial feature map can be stitched together to obtain a 512*512 six-channel third fused feature.

[0143] The second fusion feature is input into the encoding layer. The encoding layer can be used to compress data and reduce dimensionality, removing redundant information from the second fusion feature, effectively reducing the amount of data and computation.

[0144] The output data of the encoding layer is used as the input of the first convolutional layer. After convolution operation in the first convolutional layer (1×1conv), the first matrix Q is obtained.

[0145] The third fused feature is input into the second convolutional layer (1×1 conv), and after convolution, the second matrix K is obtained. The third fused feature is input into the third convolutional layer (1×1 conv), and after convolution, the third matrix V is obtained.

[0146] Multiplying the first matrix Q and the second matrix K yields the optical flow features. These features describe the motion information (how each pixel on a face moves, deforms, and distorts) between two consecutive frames, i.e., how facial expressions change.

[0147] Multiplying the optical flow features by the third matrix V yields the features corresponding to the target driving image. Since the optical flow features include motion information of the face pixels in the source image, and the third matrix V includes the first facial features of the image to be driven, the optical flow features can be used to influence the first facial features of the face to be driven. This allows the image to be driven to reproduce the motion of the pixels in the source image, thus achieving face driving. In some embodiments, the cross-attention mechanism network may further include an activation layer. The features corresponding to the target driving image are input into the activation layer, and the target driving image is output after passing through the activation layer.

[0148] In this embodiment, a cross-attention mechanism network is used to analyze key point feature maps, depth feature maps, and facial features. Figure Three Cross-alignment and comparison of different semantic spaces, and interaction and transmission of feature information and spatial location information at different pixel positions, can improve the accuracy of image understanding and processing, better explore the correlation between face and 3D information, and achieve more refined capture of facial expressions, reflecting more facial details.

[0149] In optional embodiments, such as Figure 7 As shown, step S614 may further include:

[0150] Step S700: Determine the dimension value based on the first matrix and the second matrix.

[0151] Step S702: Obtain the target driving image based on the optical flow features, the third matrix, and the dimension value.

[0152] In this embodiment, the dimension value d can be determined based on the number of columns in the first matrix Q and the second matrix K. k Then, based on matrices Q, K, V, and the dimension value d... k The target-driven graph is obtained through network computation using a cross-attention mechanism.

[0153] The calculation formula for the cross-attention mechanism is as follows:

[0154]

[0155] Where, dk It is the number of columns in the Q and K matrices.

[0156] To facilitate model optimization and improve network stability, the calculation can be divided by d. k This prevents the dot product / inner product from becoming too large and improves computational efficiency.

[0157] To make this application easier to understand, the following is combined with... Figures 8A to 8C An exemplary application is provided. In this exemplary application, service platform 2 connects to the client via a network and provides the client with face-driven services.

[0158] S11, obtain the image to be driven and the driving source image.

[0159] S12, the first facial features of the image to be driven are obtained through the first facial feature extraction network.

[0160] S13, the second facial features of the driving source image are obtained through the second facial feature extraction network.

[0161] S14: Obtain 3D point cloud information of key points in the driving source image through a 3D model.

[0162] S15, perform feature fusion on the 3D point cloud information of key points and the second facial features to obtain the first fused feature.

[0163] S16, input the first fusion feature into the facial depth network, and obtain the facial depth map through the facial depth network.

[0164] S17, feature fusion is performed on the 3D point cloud information of the face depth map and key points to obtain the second fused feature.

[0165] S18, the second fused feature is input into the encoding layer, and after convolution operation in the first convolution layer, the first matrix Q is obtained.

[0166] S19, perform feature fusion on the first facial feature and the second facial feature to obtain the third fused feature.

[0167] S20, the third fusion feature is input into the second convolutional layer, and after the convolution operation, the second matrix K is obtained.

[0168] S21, the third fusion feature is input into the third convolutional layer, and after the convolution operation, the third matrix V is obtained.

[0169] S22, multiply the first matrix Q and the second matrix K to obtain the optical flow map (optical flow characteristics).

[0170] S23, multiply the optical flow map and the third matrix V to obtain the target driving map.

[0171] S24, output the target driving graph.

[0172] In this exemplary application, the depth features of the driving source image are obtained using the second facial features and key point features of the driving source image. During the process of driving the image to be driven using the driving source image, the key point features, depth features, and second facial features of the driving source image can be combined to influence the first facial features of the image to be driven, thereby achieving precise synchronization of actions, expressions, etc. Because the depth features are used to accurately analyze the three-dimensional facial structure of the driving source image, richer facial information is provided for face driving, effectively improving the performance and accuracy of face driving.

[0173] Example 2

[0174] Figure 9 The diagram schematically illustrates a face-driving device 1000 according to Embodiment 2 of this application. This face-driving device 1000 can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of this application. The program module referred to in this embodiment is a series of computer program instruction segments capable of performing a specific function. The following description will specifically introduce the function of each program module in this embodiment. For example... Figure 9 As shown, the face-driving device 1000 may include: a first acquisition module 1100, a second acquisition module 1200, a third acquisition module 1300, and an output module 1400, wherein:

[0175] The first acquisition module 1100 is used to acquire the first facial features of the image to be driven;

[0176] The second acquisition module 1200 is used to acquire the second facial features and key point features of the driving source image; wherein, the driving source image is used to drive the image to be driven to form a target driving image;

[0177] The third acquisition module 1300 is used to acquire depth features based on the key point features and the second facial features;

[0178] The output module 1400 is used to input the first facial feature, the second facial feature, the key point feature and the depth feature into a pre-trained target neural network, and output the target driven image through the target neural network.

[0179] As an optional embodiment, the second acquisition module 1200 is further configured to:

[0180] The 3D point cloud information of key points in the driving source image is obtained by using a pre-trained 3D model;

[0181] The 3D point cloud information includes translation parameters and rotation parameters; the translation parameters represent the distance the key point moves in the three-dimensional coordinate system; and the rotation parameters represent the rotation angle of the key point in the three-dimensional coordinate system.

[0182] As an optional embodiment, the third acquisition module 1300 is further configured to:

[0183] Based on the key point features and the second facial features, a first fusion feature is obtained;

[0184] The first fused feature is input into a pre-trained facial depth network to obtain the depth feature.

[0185] As an optional embodiment, the third acquisition module 1300 is further configured to:

[0186] The key point features are loaded into the second facial features through modulation and demodulation to obtain the first fused features.

[0187] As an optional embodiment, the target neural network includes a cross-attention mechanism network.

[0188] As an optional embodiment, the cross-attention mechanism network includes an encoding layer, a first convolutional layer, a second convolutional layer, and a third convolutional layer; correspondingly, the output module 1400 is further configured to:

[0189] Based on the key point features and the depth features, a second fusion feature is obtained;

[0190] The second fusion feature is input into the coding layer;

[0191] The output data of the encoding layer is convolved through the first convolutional layer to obtain the first matrix;

[0192] Based on the first facial feature and the second facial feature, a third fusion feature is obtained;

[0193] The third fusion feature is input into the second convolutional layer to obtain the second matrix;

[0194] The third fusion feature is input into the third convolutional layer to obtain the third matrix;

[0195] Optical flow features are obtained based on the first matrix and the second matrix;

[0196] The target driving image is obtained based on the optical flow features and the third matrix.

[0197] As an optional embodiment, the output module 1400 is further configured to:

[0198] Determine the dimension value based on the first matrix and the second matrix;

[0199] The target-driven image is obtained based on the optical flow features, the third matrix, and the dimension value.

[0200] Example 3

[0201] Figure 10 This illustration schematically shows a hardware architecture diagram of a computer device 10000 suitable for implementing a face-driven method according to Embodiment 3 of this application. In some embodiments, the computer device 10000 may be a terminal device such as a smartphone, wearable device, tablet computer, personal computer, in-vehicle terminal, game console, virtual device, workbench, digital assistant, set-top box, robot, etc. In other embodiments, the computer device 10000 may be a rack server, blade server, tower server, or cabinet server (including independent servers or server clusters composed of multiple servers), etc. Figure 10 As shown, the computer device 10000 includes, but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate and be linked with each other via a system bus. Wherein:

[0202] The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of a computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as the program code for the face-driving method. In addition, the memory 10010 can also be used to temporarily store various types of data that have been output or will be output.

[0203] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.

[0204] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 10000 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communications (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0205] It should be pointed out that, Figure 10 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0206] In this embodiment, the face-driving method stored in memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of this application.

[0207] Example 4

[0208] This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the face-driving method in the embodiments.

[0209] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEP ROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the face driving method in this embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0210] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computer devices. They can be centralized on a single computer device or distributed across a network of multiple computer devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computer device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0211] It should be noted that the above are merely preferred embodiments of this application and do not limit the scope of patent protection of this application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A face-driven method, characterized in that, The method includes: Obtain the first facial features of the image to be driven; Obtain the second facial features and key point features of the driving source image; Depth features are obtained based on the key point features and the second facial features; wherein, the second facial features include overall facial feature information and specific facial feature information of the driving source image; The first facial feature, the second facial feature, the key point feature, and the depth feature are input into a pre-trained target neural network, and the target driving image is output through the target neural network; wherein, the target driving image is the image formed by the image to be driven under the driving source image; The step of obtaining depth features based on the key point feature map and the second facial features includes: Based on the key point features and the second facial features, a first fusion feature is obtained; The first fused feature is input into a pre-trained facial depth network to obtain the depth feature.

2. The method according to claim 1, characterized in that, The acquisition of the second facial features and key point features of the driving source image includes: The 3D point cloud information of key points in the driving source image is obtained by using a pre-trained 3D model; The 3D point cloud information includes translation parameters and rotation parameters; the translation parameters represent the distance the key point moves in the three-dimensional coordinate system; and the rotation parameters represent the rotation angle of the key point in the three-dimensional coordinate system.

3. The method according to claim 1, characterized in that, The step of obtaining the first fusion feature based on the key point features and the second facial features includes: The key point features are loaded into the second facial features through modulation and demodulation to obtain the first fused features.

4. The method according to any one of claims 1 to 3, characterized in that, The target neural network includes a cross-attention mechanism network.

5. The method according to claim 4, characterized in that, The cross-attention mechanism network includes an encoding layer, a first convolutional layer, a second convolutional layer, and a third convolutional layer; correspondingly, the step of outputting the target-driven image through the target neural network includes: Based on the key point features and the depth features, a second fusion feature is obtained; The second fusion feature is input into the coding layer; The output data of the encoding layer is convolved through the first convolutional layer to obtain the first matrix; Based on the first facial feature and the second facial feature, a third fusion feature is obtained; The third fusion feature is input into the second convolutional layer to obtain the second matrix; The third fusion feature is input into the third convolutional layer to obtain the third matrix; Optical flow features are obtained based on the first matrix and the second matrix; The target driving image is obtained based on the optical flow features and the third matrix.

6. The method according to claim 5, characterized in that, The target-driven image is obtained based on the optical flow features and the third matrix, including: Determine the dimension value based on the first matrix and the second matrix; The target-driven image is obtained based on the optical flow features, the third matrix, and the dimension value.

7. A face-driving device, characterized in that, The device includes: The first acquisition module is used to acquire the first facial features of the image to be driven; The second acquisition module is used to acquire the second facial features and key point features of the driving source image; wherein, the driving source image is used to drive the image to be driven to form a target driving image; The third acquisition module is used to acquire depth features based on the key point features and the second facial features; wherein, the second facial features include overall facial feature information and specific facial feature information of the driving source image; The output module is used to input the first facial feature, the second facial feature, the key point feature and the depth feature into the pre-trained target neural network, and output the target driven image through the target neural network; The third acquisition module is further configured to acquire a first fusion feature based on the key point features and the second facial features; and input the first fusion feature into a pre-trained facial depth network to acquire the depth features.

8. A computer device, characterized in that, include: At least one processor; and A memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 6.

Images