A method and electronic device for optimizing face driving parameters

Abstract

This application relates to the field of 3D reconstruction technology, providing a method and electronic device for optimizing face driving parameters. Applied to a remote 3D communication system, before remote 3D interaction, the gender and age identified in the initial face image of the target object are used as prior knowledge. Corresponding wrinkle information is loaded and added to the initial parameterized head model, increasing the detail description of facial wrinkles and improving the accuracy of face reconstruction. During remote 3D interaction, the wrinkle information is used to optimize the parameters used to drive the initial parameterized head model based on the target object's current facial expression. This ensures that each expression base of the target object contains descriptive features of wrinkles, further improving the driving accuracy of facial expressions by adding wrinkles, thereby better representing the detail accuracy of facial expression driving and enhancing the realism of facial expression expression.

Description

Technical Field

[0001] This application relates to the field of 3D reconstruction technology, and provides a method and electronic device for optimizing face driving parameters. Background Technology

[0002] 3D digital humans form the foundation of remote 3D communication systems. By placing the 3D digital humans of both interacting parties in the same virtual environment, an immersive, face-to-face interaction is achieved. During human-to-human interaction, basic attribute information such as gender, age, and skin color can be obtained through a person's facial features, and emotional changes can be discerned through rich facial postures and expressions. Therefore, facial reconstruction is one of the most important parts of human body reconstruction in remote 3D communication systems.

[0003] In remote 3D communication systems, to ensure smooth interaction, a parametric head model is typically used to reconstruct the 3D face. This method performs well under large facial movements, such as opening the mouth, raising eyebrows, and smiling, as well as rigid movements like head rotation and translation. However, because the driving parameters of the parametric head model ignore detailed facial information, such as nasolabial folds and crow's feet when smiling, and forehead wrinkles when raising eyebrows, the reconstructed 3D facial model has low precision. This makes it difficult for both parties to understand each other's expressions during remote 3D communication, reducing the immersive experience. Summary of the Invention

[0004] This application provides a method and electronic device for optimizing face driving parameters to improve the driving accuracy of three-dimensional digital human facial expressions.

[0005] On one hand, embodiments of this application provide a method for optimizing face-driving parameters, applied to a remote 3D communication system, the method comprising:

[0006] Before remote 3D interaction, an initial facial image of the target object is acquired, and initial facial key points are extracted from the initial facial image, as well as the gender and age of the target object are identified.

[0007] Based on the gender and age of the target object, load the corresponding wrinkle information;

[0008] Based on the wrinkle information, a standard parametric head model is obtained;

[0009] Based on the initial facial key points and the standard parametric head model, determine the initial parametric head model of the target object;

[0010] During remote 3D interaction, a target face image of the target object is acquired, and key points of the target face are extracted from the target face image;

[0011] Based on the initial parameterized head model and the target face key points, determine the target driving parameters;

[0012] The target driving parameters are optimized based on the wrinkle information, so that the rendering display terminal drives the pre-stored initial parameterized head model to move according to the optimized target driving parameters, thereby obtaining the target parameterized head model of the target object.

[0013] On the other hand, embodiments of this application provide an electronic device for use in a remote three-dimensional communication system, including a processor, a memory, and a communication interface, wherein the memory, the communication interface, and the processor are connected via a bus;

[0014] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0015] Before remote 3D interaction, an initial facial image of the target object is acquired, and initial facial key points are extracted from the initial facial image, as well as the gender and age of the target object are identified.

[0016] Based on the gender and age of the target object, load the corresponding wrinkle information;

[0017] Based on the wrinkle information, a standard parametric head model is obtained;

[0018] Based on the initial facial key points and the standard parametric head model, determine the initial parametric head model of the target object;

[0019] During remote 3D interaction, a target face image of the target object is acquired, and key points of the target face are extracted from the target face image;

[0020] Based on the initial parameterized head model and the target face key points, determine the target driving parameters;

[0021] The target driving parameters are optimized based on the wrinkle information, and the optimized target driving parameters are sent to the rendering and display terminal through the communication interface, so that the rendering and display terminal drives the pre-stored initial parameterized head model to move according to the optimized target driving parameters, thereby obtaining the target parameterized head model of the target object.

[0022] On the other hand, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the method steps for optimizing face-driving parameters provided in embodiments of this application.

[0023] The method and electronic device for optimizing face driving parameters provided in this application are applied to a remote 3D communication system. Before remote 3D interaction, the gender and age identified in the initial face image of the target object are used as prior knowledge, and corresponding wrinkle information is loaded and added to the initial parameterized head model to increase the detail description of facial wrinkles and improve the accuracy of face reconstruction. During remote 3D interaction, the wrinkle information is used to optimize the parameters used to drive the initial parameterized head model according to the current facial expression of the target object, so that each expression base of the target object contains the descriptive features of wrinkles. By adding wrinkles, the driving accuracy of facial expressions is further improved, thereby better representing the detail accuracy of the reconstructed face and improving the realism of face reconstruction. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This application provides an architecture diagram of a remote 3D interactive system.

[0026] Figure 2 This application provides an architecture diagram of another remote 3D interactive system for embodiments of the present application;

[0027] Figure 3 A schematic diagram illustrating a method for optimizing driving parameters provided in an embodiment of this application;

[0028] Figure 4 A schematic diagram illustrating another method for optimizing driving parameters provided in an embodiment of this application;

[0029] Figure 5 This is a schematic diagram illustrating the method of adding wrinkle information before remote three-dimensional interaction provided in an embodiment of this application;

[0030] Figure 6 This is a schematic diagram illustrating the method of adding wrinkle information during remote three-dimensional interaction provided in an embodiment of this application;

[0031] Figure 7 A flowchart illustrating the method for optimizing face-driving parameters provided in this application embodiment;

[0032] Figure 8 This is a schematic diagram of facial wrinkle types provided in an embodiment of this application;

[0033] Figure 9 This application provides schematic diagrams of common facial wrinkles in its embodiments.

[0034] Figure 10 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0036] Research has found that for humans, emotional expression equals 7% words + 38% vocalizations + 55% facial expressions, highlighting the importance of facial expressions. Furthermore, although only 43 facial muscles drive facial movements, they can express tens of thousands of emotions, making it a significant subject of research in psychology, art, philosophy, and other fields.

[0037] As an important part of the emotional expression of the three-dimensional digital human presented in a remote three-dimensional communication system, the level of detail of the face directly affects the remote three-dimensional interactive experience.

[0038] Currently, reconstructed 3D facial models are mainly divided into three types: procedural parametric models, physiological structure models, and data-driven parametric models. Among them:

[0039] Procedural parametric models primarily describe changes in facial appearance and shape through facial construction parameters and facial animation parameters, with the MPEG-4 facial animation standard having the greatest impact. This standard uses static and dynamic parameters to represent changes in facial shape among different individuals and with different expressions. However, this model only simulates facial movement through simple surface geometry, resulting in poor realism and a lack of consideration for physiological movement mechanisms.

[0040] Physiological structural models construct skeletal and muscular constraints by referencing facial anatomy and simulating facial physiology. The most famous example is the Facial Action Coding System (FACS). FACS decomposes facial expressions into 46 basic action units, each representing the contraction or relaxation of a set of muscles, and has been used in various facial animation studies. However, limited by anatomical accuracy and computational complexity, physiological structural models require significant computation and interaction, making them unsuitable for real-time facial animation.

[0041] Data-driven parametric models provide a compact and simple method for representing faces. This method takes into account that although there are subtle differences in the faces of different individuals, the general trends and processes of change are basically similar. Therefore, it uses existing face data to build a parametric head model, and then uses different parameters to generate faces with different shapes, expressions, and appearances.

[0042] During remote 3D communication, the real-time, high-precision, and efficient model-driven and rendering display effects on virtual reality devices directly impact the user's immersive experience. To ensure a face-to-face immersive experience, real-time, efficient, and high-precision face reconstruction has become one of the core elements of 3D digital human reconstruction. However, high-precision 3D face models often mean a large amount of data, which can conflict with the real-time rendering display of virtual reality devices.

[0043] Data-driven parametric models (DPPs) can generate personalized faces by inputting only a small number of driving parameters, such as expression, pose, and shape parameters. Combined with pre-stored texture parameters, this allows for the generation of high-precision faces. This reconstruction method better balances the conflict between model accuracy and real-time communication, making it the most commonly used reconstruction method in remote 3D communication systems.

[0044] Currently, the main problems encountered when reconstructing the face of a digital human using a data-driven parametric model are as follows:

[0045] I. Data Collection Issues

[0046] During remote 3D communication, various face data acquisition problems may occur, such as occlusion, lighting, texture, rapid motion blur, or single-view limitation, resulting in incomplete face data. As a result, when driving the parameterized head model, the driving parameters cannot be accurately calculated, which can easily lead to low reconstruction accuracy or delay.

[0047] II. Reconstruction Accuracy Issues

[0048] With limited data input, there is a high demand on the driving algorithm. The execution efficiency and output accuracy of the driving algorithm directly affect the display effect presented on the virtual reality device.

[0049] Currently, in face reconstruction, using parameter-driven parametric head models yields good reconstruction results under significant facial movements, such as opening the mouth, raising eyebrows, and smiling, as well as rigid movements like head rotation and translation. However, because the driving parameters ignore detailed facial information, such as nasolabial folds and crow's feet when smiling, and forehead wrinkles when raising eyebrows, the reconstructed 3D facial model has lower precision. This makes it difficult for both parties to understand each other's expressions during remote 3D communication, reducing the immersive experience.

[0050] To improve the detail accuracy of facial reconstruction using data-driven models, one approach involves adding depth data acquisition to the acquired RGB images to provide higher-quality driving parameters. However, depth data acquisition increases the cost of the acquisition equipment and requires higher quality RGB images. Another approach uses linear combinations of expression bases to drive the face, but these bases cannot express details such as wrinkles. Yet another approach uses local detail regressors to express wrinkle information, but this does not consider the specificities of wrinkles caused by gender and age.

[0051] Considering that the computational power, accuracy, and data transmission latency of face reconstruction significantly impact the user's immersive experience, and that facial wrinkle details can enhance the expressiveness of the reconstructed face, thereby helping both parties in the interaction to better understand each other's expressions, wrinkle details are essential in remote 3D communication systems that require high reconstruction accuracy.

[0052] Wrinkles are usually directly related to a person's age, gender, and other factors.

[0053] At the age level, wrinkles are the result of the combined effects of physiological tissues and environmental factors as we age, and the number and depth of wrinkles are linearly proportional to age. Among these, internal tissue and organ aging factors are particularly prominent in the wrinkle formation process, such as changes in the flattening of the dermal-epidermal junction, dermal-epidermal atrophy, reduction of collagen and reticulin, and loss of fibroblasts. As we age, wrinkles on the face gradually increase, and the corresponding facial details in the model should also increase.

[0054] At the gender level, wrinkles differ between men and women. Studies show that men have a greater number of sebaceous glands, sweat glands, blood vessels, and interstitial connections in their perioral skin than women. Although there is no significant difference in the number of hair follicles in men's and women's skin, the average number of sebaceous glands per hair follicle is much higher in men than in women. Therefore, women exhibit more and deeper perioral wrinkles. Furthermore, women's skin contains fewer skin appendages than men's skin, resulting in several distinguishable types of deep wrinkles on the face, the most common being linear wrinkles, including but not limited to forehead wrinkles, crow's feet, nasolabial folds, and perioral wrinkles. This also explains why vertical perioral wrinkles are more common in women than in men.

[0055] In view of this, embodiments of this application provide a method and apparatus for optimizing face driving parameters. The recognized gender and age are used as prior knowledge of the face. During the initial reconstruction and real-time driving process, based on the mapping relationship between gender and age and wrinkles, matching wrinkle information is loaded for the corresponding digital human. The wrinkles are used to optimize the driving parameters of the parameterized head model, thereby improving the accuracy and realism of the face reconstructed based on the parameterized head model, and thus enhancing the user's immersive experience.

[0056] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0057] The core of the remote 3D communication system involves real-time 3D reconstruction technology, encoding, decoding and transmission technology of 3D data (such as shape, motion and material data), and immersive VR / AR rendering and display technology. Among these, the transmission of parameters used to drive the parametric head model has a significant impact on the immersive experience of the rendering and display terminal user.

[0058] See Figure 1 This is an architecture diagram of a remote three-dimensional communication system provided in an embodiment of this application. The system includes a data acquisition terminal 100, a transmission terminal 200, and a rendering and display terminal 300.

[0059] The acquisition terminal 100 is mainly used for 3D reconstruction based on the acquired images, and includes an acquisition camera 101 and a graphics workstation 102. The acquisition camera 101 can be a regular RGB camera or an RGBD camera, used to acquire image data. The graphics workstation 102 is used to extract the driving parameters and texture parameters of the face from the image data, and uses the gender and age identified in the image data as prior knowledge to optimize the driving parameters of the parameterized head model. The optimized driving parameters and texture parameters are then uploaded to the transmission terminal 200. The optimized driving parameters and texture parameters are transmitted separately.

[0060] After receiving the driving parameters and texture parameters sent by the acquisition terminal 100, the transmission terminal 200 encodes and decodes them, and then transmits them to the rendering and display terminal 300.

[0061] The rendering and display terminal 300 drives the parametric head model to move according to the acquired driving parameters and renders and displays the head of the digital human in the virtual scene. The texture parameters are acquired and stored locally during the initial modeling process. This ensures that during real-time remote 3D interaction, only driving parameters with a small data volume are received, thus guaranteeing the rendering and display efficiency of the rendering and display terminal 300.

[0062] In one optional implementation, the acquisition terminal 100 and the rendering display terminal 300 in this system architecture can be AR products or VR products. The transmission terminal 200 is a server, which can be a single server, a server cluster, or a cloud server with cloud storage, cloud computing, and cloud processing capabilities.

[0063] It should be noted that in the embodiments of this application, the acquisition terminal 100 and the rendering display terminal 300 are user-defined. For example, the VR glasses worn by user A are the acquisition terminal 100 for user B, and the VR glasses worn by user B are the rendering display terminal 300 for user A. Similarly, the VR glasses worn by user B are the acquisition terminal 100 for user A, and the glasses worn by user A are the rendering display terminal 300 for user B.

[0064] Optionally, during remote 3D interaction, the rendering display terminal 300 displays the digital person and virtual scene on connected 2D terminals such as mobile phones, TVs, and tablets.

[0065] In the embodiments of this application, in the remote three-dimensional communication system, the user corresponding to the acquisition terminal 100 can view the environmental information of the rendering display terminal through XR glasses (such as VR glasses, AR glasses, etc.), that is, the user corresponding to the acquisition terminal 100 and the rendering display terminal 300 is in the same virtual environment.

[0066] Optionally, the three-terminal devices in the remote three-dimensional communication system in this embodiment can be deployed separately according to the actual scenario requirements.

[0067] For example, in a virtual live streaming scenario, the host end is equipped with the system's acquisition terminal 100, which is used to transmit the data of the reconstructed 3D digital human of the host to the cloud. The user end is equipped with a rendering and display terminal 300, so that users can browse the live streaming content through AR glasses, mobile phones, TVs and other devices.

[0068] For example, in a virtual meeting scenario, two meeting rooms in a remote meeting are simultaneously equipped with a data acquisition terminal 100 and a rendering display terminal 300, which are used to send the 3D reconstruction data of the local user to the other end and to display the 3D digital human of the other end user, thereby enabling real-time remote 3D communication between the two meeting rooms in different locations.

[0069] Optionally, in some embodiments, the data processing of the acquisition terminal 100 and the rendering display terminal 300 in the remote 3D communication system can be modified according to their computing capabilities.

[0070] For example, such as Figure 2 The diagram shown is an architecture diagram of another remote three-dimensional communication system provided in an embodiment of this application. The system includes a data acquisition terminal 100, a transmission terminal 200, and a rendering and display terminal 300.

[0071] The acquisition terminal 100 includes an acquisition camera 101, which is mainly used to acquire 3D reconstructed images and upload them to the transmission terminal 200. The acquisition camera 101 can be a standard RGB camera or an RGBD camera, used to acquire image data.

[0072] After decoding the image data sent by the acquisition terminal 100, the transmission end 200 extracts the driving parameters and texture parameters for face reconstruction from the image data. Using the gender and age identified in the image data as prior knowledge, it optimizes the driving parameters of the parameterized head model. The optimized driving parameters and texture parameters are then encoded and transmitted to the rendering and display terminal 300. The optimized driving parameters and texture parameters are transmitted separately.

[0073] The rendering and display terminal 300 drives the parametric head model to move according to the acquired driving parameters and renders and displays the head of the digital human in the virtual scene. The texture parameters are acquired and stored locally during the initial modeling process. This ensures that during real-time remote 3D interaction, only driving parameters with a small data volume are received, thus guaranteeing the rendering and display efficiency of the rendering and display terminal 300.

[0074] exist Figure 2 In the system shown, the acquisition terminal 100 is only responsible for the acquisition and transmission of image data, while the transmission terminal 200 performs large-scale calculations such as data processing and reconstruction. In this way, the acquisition terminal 100, which has poor computing power, can reduce its computing load, save energy, avoid the problem of users feeling uncomfortable due to excessive heat, and at the same time ensure the real-time performance of remote three-dimensional communication.

[0075] Optionally, for rendering and display terminals 300 with poor computing power, the data processing part before rendering and display can also be moved to the transmission end 200. The transmission end 200 can directly calculate each frame of the rendering and display terminal 300 during remote 3D communication, thereby reducing the computing load of the rendering and display terminal 300 and improving the rendering and display efficiency.

[0076] For example, the transmission terminal 200 uses texture parameters and optimized driving parameters to drive the movement of the parameterized head model, and sends the vertex and face data of the driven parameterized head model to the rendering and display terminal 300. The rendering and display terminal 300 directly renders the head of the 3D digital human and the virtual scene based on the acquired data.

[0077] The remote 3D communication system provided in this application mainly analyzes, reconstructs, and drives the data collected by the acquisition terminal 100. This application does not impose any limiting requirements on whether the computing power is placed on the acquisition terminal, the transmission terminal, or the rendering and display terminal.

[0078] Whether Figure 1 The remote 3D reconstruction system shown is still Figure 2 The remote 3D reconstruction system shown calculates and optimizes the driving parameters for moving the parametric head model, which directly affects the accuracy and realism of the digital face. The driving parameters play a crucial role in the accuracy of face reconstruction. Therefore, the method for optimizing face driving parameters provided in this application mainly focuses on optimizing the driving parameters in the parametric head model.

[0079] See Figure 3 This diagram illustrates the optimization method of driving parameters provided in this application embodiment. Based on image data collected by the acquisition terminal 100, facial recognition and segmentation are performed to obtain a face image. Facial feature points are extracted from the face image, and based on the extracted facial feature points, initial driving parameters of the face, such as shape parameters, pose parameters, and expression parameters, are calculated. At the same time, gender recognition and age recognition are performed based on the face image, and gender and age are used as prior knowledge to optimize the initial driving parameters. Based on the optimized driving parameters, the parameterized head model is driven to move, thereby improving the driving accuracy of facial expressions and making the reconstructed face have wrinkle details, thus improving the reconstruction accuracy of the face.

[0080] Optionally, in some embodiments, the acquisition terminal 100 further includes a microphone for acquiring voice data during remote 3D interaction. For example... Figure 4 As shown, the voice data is used to optimize the gender and age identified in the face image, thereby improving the accuracy of gender and age recognition, which in turn improves the driving precision of the driving parameters, and further enhances the realism and detail features of the face reconstruction.

[0081] Optionally, the parametric head model in this embodiment can be constructed based on the FLAME model, which consists of two parts: a standard linear blend skinning (LBS) and a blend shape. The standard mesh model used has N = 5023 vertices and K = 4 joints (located in the neck, jaw, and two eyeballs, respectively). The main components of the FLAME parametric head model are as follows:

[0082]

[0083]

[0084] in, Indicates shape parameters, Represents attitude parameters (including motion parameters of the skeleton), For expression parameters. A vertex coordinate can uniquely identify the 3D geometric model of the head. W() represents a linear skinning function used to transform the head model mesh T along the joints, J() represents a function to predict the positions of different head joint points, T represents the head model mesh, and B... s () represents the influence function of shape parameters on the head model mesh T, B p () represents the function that influences the attitude parameters on the head model mesh T, B e () represents the function that influences facial expression parameters on the head model mesh T, where T p () represents the function that deforms the head model mesh T under the combined effects of shape, pose, and expression parameters. s, p, e, and ω represent the shape weight, pose weight, expression weight, and skinning weight, respectively. s, p, e, and ω are obtained through training on pre-constructed head sample data. After training s, p, e, and ω, subsequent steps only require providing... By using the same parameters, a topologically consistent parameterized human head model can be obtained.

[0085] In the embodiments of this application, before remote 3D interaction, an initial parameterized head model of both parties in a natural state (i.e., expressionless state) is pre-constructed and stored in the other party's device. In this way, during remote 3D interaction, the vertex movement on the initial parameterized head model can be driven by real-time target driving parameters (including target shape parameters, target pose parameters and target expression parameters) to obtain a target parameterized head model that reflects the real faces of both parties.

[0086] Considering that the topological structure of the parametric head model is relatively simple and cannot represent detailed wrinkles, this embodiment incorporates gender and age as prior knowledge during the initial reconstruction of the parametric head model and the real-time driving of the target parametric head model. By loading the wrinkle information corresponding to the prior knowledge, the driving parameters of the parametric head model are optimized, thereby improving the accuracy of facial driving and enhancing the realism and detail accuracy of face reconstruction.

[0087] The process of reconstructing the initial parametric head model before remote 3D interaction is as follows: Figure 5 As shown, the gender and age of the target object are used as prior knowledge, and corresponding wrinkle information is loaded. The wrinkle information is then used to process the standard parametric head model. The point cloud formed by the initial facial key points extracted from the face image in its natural state is matched with the vertices in the processed standard parametric head model. Based on the matching results, the Singular Value Decomposition (SVD) algorithm and Cholesky decomposition method are used to obtain the initial shape parameters, initial expression parameters, and initial pose parameters that can drive the deformation of the standard parametric head model, thereby obtaining the initial parametric head model of the target object in its natural state.

[0088] See Figure 6 This paper describes the real-time driving process of the initial parameterized head model during remote 3D interaction. The point cloud formed by extracting key points of the target face from the real-time acquired target face image is matched with the vertices in the stored initial parameterized head model. Based on the matching results, the SVD algorithm and Cholesky decomposition method are used to solve for the target shape parameters, target expression parameters, and target pose parameters that can drive the deformation of the initial parameterized head model. Then, the gender and age of the target object are used as prior knowledge to load the corresponding wrinkle information, and the wrinkle information is used to optimize the target shape parameters, target expression parameters, and target pose parameters, so as to obtain a target parameterized head model with consistent topology and capable of expressing wrinkle details.

[0089] See Figure 7 This is a flowchart of a method for optimizing face-driving parameters provided in an embodiment of this application. This process is executed by an electronic device, which may be... Figure 1 The data acquisition terminal 100 in the middle can also be Figure 2 The method for using a server with a transmission terminal of 200 mainly includes the following steps:

[0090] S701: Before remote 3D interaction, acquire the initial face image of the target object, extract the initial facial key points from the initial face image, and identify the gender and age of the target object.

[0091] For the target object of the acquisition terminal, the front RGB image of the target object is acquired by an RGB camera, and the face recognition of the front RGB image of the target object is performed. The face is then segmented according to a specific ratio (such as a 256*256 face segmentation range) in order to select the face and obtain the initial face image, thereby reducing the interference of non-face areas.

[0092] After obtaining the initial face of the target object, the initial facial key points are extracted from the target face image to obtain the initial point cloud data of the facial key points.

[0093] Among these, facial recognition and segmentation technologies, as well as facial landmark extraction techniques, are relatively mature, and this application does not impose limiting requirements on them. For example, facial recognition and segmentation can employ lightweight deep learning algorithms (such as ResNet), machine learning algorithms (such as Histogram of Oriented Gradients (HOG), convolutional neural network algorithms, etc. Facial landmark extraction can utilize OpenCV's Dlib library to extract 68 facial landmarks, or it can employ the PracticalFacial Landmark Detector algorithm, or the MediapipeFacemesh algorithm to extract 468 dense facial landmarks. The more facial landmarks used, the higher the level of detail in the facial representation. However, regardless of the algorithm used, facial recognition and segmentation must meet the requirements of real-time performance, efficiency, and stability for remote 3D interaction to ensure the effectiveness of subsequent optimization and rendering.

[0094] In the embodiments of this application, the driving parameters of the parameterized head model are optimized through facial wrinkles to represent the wrinkle details of the digital human face. This requires obtaining the wrinkle information of the target object. Considering that wrinkle information is generally related to attributes such as gender and age, gender and age recognition is also required after obtaining the face image of the target object.

[0095] Gender recognition can be viewed as a complex, large-scale quadratic pattern classification problem, where the classifier categorizes input data into male and female. Currently, gender recognition methods mainly fall into three categories: face-based gender recognition algorithms, Fisher's criterion-based gender recognition methods, and Adaboost+SVM-based face gender recognition algorithms. Considering that gender is primarily affected by beauty filters and makeup, but data collection is relatively easy, a large number of gender training samples can be used to train a classification model, obtaining male and female gender characteristics. Based on the trained classification model, the gender of the target object can then be determined.

[0096] Age recognition can be based on deep learning methods, treating age estimation as a classification problem, a regression problem, or a ranking problem. Multi-class age estimation completely ignores the order information of age labels, age regression is oversimplified into a linear model, and age ranking uses order information related to age, transforming age estimation into a ranking problem. This application's embodiment uses a ranking approach for age recognition, analyzing a series of binary classification results to obtain the predicted age.

[0097] Optionally, when identifying age based on the ranking approach, age features can be learned independently for different age groups to ensure that the learned age features have effective representation capabilities, such as by using the Ranking-CNN algorithm.

[0098] S702: Load the corresponding wrinkle information based on the target object's gender and age.

[0099] Optionally, wrinkle information may include at least one of the following: wrinkle number, wrinkle density, wrinkle length, and wrinkle type.

[0100] There are many types of facial wrinkles, for example, such as... Figure 8 As shown, wrinkles from the forehead to the neck mainly include forehead wrinkles, frown lines, crow's feet wrinkles, tear trough wrinkles, nasolabial folds, expression lines, lip lines, chin lines, and neck wrinkles. Among these, the most common are forehead wrinkles, crow's feet wrinkles, nasolabial folds, and marionette lines. Figure 9 As shown.

[0101] Considering that wrinkles are related to age and gender, and that wrinkle types, numbers, densities, and lengths vary between different genders and individuals, this application pre-establishes a mapping relationship between gender, age, and wrinkle information. Based on this mapping relationship, once the gender and age of the target object are identified, the corresponding wrinkle information can be loaded.

[0102] Optionally, in some embodiments, the mapping relationship may include a first sub-mapping relationship between gender and wrinkles and a second sub-mapping relationship between age and wrinkles, or it may be a mapping relationship between gender and age combined with wrinkles.

[0103] It should be noted that in a remote 3D interactive system, the gender and age of the same person will not change. Therefore, for the same target object, the gender and age only need to be identified once before the interaction and retained as prior knowledge. During subsequent real-time interaction, there is no need to re-identify, which saves computation and improves rendering and display efficiency.

[0104] In some embodiments, to improve the driving accuracy of facial expressions, after identifying the gender and age of the target object, the voice data of the target object is acquired by the microphone of the acquisition terminal, the gender and age of the target object are identified, and the gender identified by the voice data is weighted with the gender of the target object identified in the initial face image. Similarly, the age identified by the voice data is weighted with the age of the target object identified in the initial face image, thereby optimizing the gender and age of the target object. This allows for more accurate wrinkle information, resulting in higher accuracy of wrinkle details on the reconstructed face of the target object.

[0105] S703: Obtain a standard parametric head model based on wrinkle information.

[0106] In one alternative implementation, a corresponding standard parametric head model is selected based on the wrinkle information of the loaded target object; or, the topology of the basic parametric head model is processed based on the wrinkle information of the loaded target object to obtain a standard parametric head model.

[0107] S704: Determine the initial parametric head model of the target object based on the initial facial key points and the standard parametric head model.

[0108] Since the head model in this embodiment is constructed based on a parametric head model, the parametric head model consists of shape parameters, posture parameters, and expression parameters. The rigid movement of the head is mainly controlled by posture parameters, while facial expression changes are primarily driven by expression parameters. Appropriate expression parameters can drive accurate facial expressions.

[0109] Shape parameters, pose parameters, and expression parameters can be viewed as driving parameters for deforming facial expressions. The calculation process of these driving parameters mainly involves extracting initial facial feature points from the initial face image and obtaining a standard parametric head model. These are then fitted and optimized using a nonlinear least squares method to obtain the initial shape parameters, initial pose parameters, and initial expression parameters. By calculating these initial shape parameters, initial pose parameters, and initial expression parameters, the various expression bases of the target object's face in its natural state can be obtained.

[0110] Among them, specific solution methods for nonlinear least squares include, but are not limited to, the Gauss-Newton algorithm, the Levenberg-Marquardt algorithm, and the SVD algorithm.

[0111] After obtaining the initial parametric head model of the target object, it is stored on the peer device, and remote 3D real-time interaction is performed based on it. In this way, real-time face reconstruction can be completed by transmitting a small number of parameters used to drive the movement of the initial parametric head model during the interaction, with relatively low transmission pressure on network bandwidth.

[0112] S705: During remote 3D interaction, acquire the target face image of the target object and extract the target face key points from the target face image.

[0113] The process of extracting the target facial landmarks is the same as that of extracting the initial facial landmarks, and will not be described again here.

[0114] S706: Determine the target driving parameters based on the initial parametric head model and the target face key points.

[0115] Similarly, in the calculation of target driving parameters, the target face feature points extracted from the target face image and the pre-stored initial parameterized head model are fitted and optimized using a nonlinear least squares method to obtain target shape parameters, target pose parameters and target expression parameters, thereby obtaining driving parameters used to drive the motion of the initial parameterized head model.

[0116] S707: Optimize the target driving parameters based on wrinkle information so that the rendering display terminal can drive the pre-stored initial parameterized head model to move according to the optimized target driving parameters, thereby obtaining the target parameterized head model of the target object.

[0117] When reconstructing faces in real time based on a data-driven model, the target driving parameters are interpolated based on the wrinkle information corresponding to gender and age to obtain optimized target driving parameters. When the optimized driving parameters are used to drive the initial parameterized head model for the current expression of the target object, the wrinkles on the reconstructed target object's face are adapted to the target object's current expression, thereby improving the detail accuracy and realism of face reconstruction.

[0118] In practice, the target shape parameters in the target driving parameters are optimized based on wrinkle information, so that the rendering display terminal drives the pre-stored initial parameterized head model to move according to the target pose parameters, target expression parameters and optimized target shape parameters, thereby obtaining the target parameterized head model of the target object.

[0119] Generally, a human face has about 51 expression bases. As facial expressions change, wrinkles appear at different expression bases, and these expression base areas show distinct striped shadows due to skin folds. For each target object, although the distribution of wrinkles varies under different expressions, the locations where the wrinkles appear remain fixed.

[0120] By linearly weighting and summing the various facial expression bases, the target driving parameters for the initial parameterized head model to represent the current expression are obtained. In other words, the optimization process of the target driving parameters is actually the process of solving for each expression base. The optimization formula is expressed as follows:

[0121]

[0122] Where base represents the initial driving parameters in the natural state (where the initial shape and expression parameters in the initial parameterized head model are 0), β i Let ψ represent the target shape parameter of the i-th expression basis under the current expression. i The target expression parameters for the i-th expression base under the current expression, where n is the total number of expression bases.

[0123] Therefore, based on the target facial key points extracted from the target facial image, the target driving parameters for facial expression driving are determined. Then, the target driving parameters are optimized according to the wrinkle information corresponding to gender and age. This allows for the control and calculation of the wrinkle appearance state and the required texture data under different actions such as smiling, opening the mouth, and grimacing, thereby increasing the wrinkle details of the model.

[0124] By optimizing the target-driven parameters based on wrinkle information corresponding to different prior knowledge, the wrinkle details of the face are expressed differently, thus better reflecting wrinkles that match the gender and age of the target object, improving the detail accuracy and realism of face reconstruction.

[0125] For example, in the wrinkle information corresponding to a 30-year-old compared to a 50-year-old, the wrinkle type, number, and density are smaller, and the wrinkle length is shorter. Therefore, in the driven target parameterized head model, the face of the 30-year-old target object has fewer wrinkles.

[0126] In the embodiments of this application, the gender and age of the target object are used as prior knowledge, and corresponding wrinkle information is loaded. The wrinkle information is added in the initial reconstruction before remote 3D interaction and in the real-time driving during the remote interaction process. In this way, based on the conventional large-scale facial expression driving, the wrinkle information is used to optimize the driving parameters of the facial expression of the initial parameterized head model determined by the facial key points, thereby distinguishing the wrinkle details corresponding to target objects of different ages and genders, increasing the detail information of facial expression, and improving the reconstruction accuracy and realism of the model.

[0127] Considering the differences in wrinkles corresponding to different expression bases under different facial expressions, in order to further improve the accuracy of driving parameter optimization, in some embodiments, after optimizing the target driving parameters using wrinkle information, the target geometric reconstruction parameters are further optimized by setting corresponding weight coefficients for the weights of each expression base, thereby adjusting the influence of wrinkles at different expression bases on the target geometric parameters, making the optimized target geometric reconstruction parameters more reasonable.

[0128] In practice, based on the optimized target driving parameters, the weight coefficients of each expression base of the target object's face are set, and the weights of each expression base are adjusted according to the weight coefficients to perform secondary optimization of the target driving parameters.

[0129] In the embodiments of this application, prior knowledge of the target object is obtained through gender and age recognition, such as {gender: male, female} and {age: 10, 20, 30, ..., 90}. Based on the prior knowledge, the corresponding wrinkle information is loaded, and the wrinkle information is used to optimize the target driving parameters used to deform the initial parameterized head model according to the current facial expression of the target object. This makes each expression base of the target object contain wrinkle description features, thereby improving the driving accuracy of facial expressions by adding wrinkles, better representing the detail accuracy of the reconstructed face, and improving the realism of the face reconstruction.

[0130] Based on the same technical concept, embodiments of this application provide an electronic device for use in a remote three-dimensional communication system. This electronic device can be... Figure 1 The data acquisition terminal in the middle can also be Figure 2 The server at the transmission end of the electronic device can implement the above-mentioned method and steps for optimizing face reconstruction parameters and achieve the same technical effect.

[0131] See Figure 10 The electronic device includes a processor 1001, a memory 1002, and a communication interface 1003, which are connected via a bus 1004.

[0132] The memory 1002 stores a computer program, and the processor 1001 performs the following operations according to the computer program:

[0133] Before remote 3D interaction, the initial facial image of the target object is acquired, and the initial facial key points are extracted from the initial facial image, as well as the gender and age of the target object are identified.

[0134] Load the corresponding wrinkle information based on the target object's gender and age;

[0135] Based on wrinkle information, a standard parametric head model is obtained;

[0136] Based on the initial facial key points and the standard parametric head model, determine the initial parametric head model of the target object;

[0137] During remote 3D interaction, acquire the target face image of the target object and extract the key points of the target face from the target face image;

[0138] Based on the initial parameterized head model and the target face key points, determine the target driving parameters;

[0139] The target driving parameters are optimized based on the wrinkle information, and the optimized target driving parameters are sent to the rendering display terminal through the communication interface 1003, so that the rendering display terminal can drive the pre-stored initial parameterized head model to move according to the optimized target driving parameters, thereby obtaining the target parameterized head model of the target object.

[0140] Optionally, before loading the corresponding wrinkle information based on the target object's gender and age, the processor 1001 also performs the following operations:

[0141] Acquire the voice data of the target object, and identify the gender and age of the target object from the voice data;

[0142] The gender of the target object identified in the voice data and the age of the target object identified in the initial face image are weighted separately to optimize the gender and age of the target object identified in the initial face image.

[0143] Optionally, wrinkle information may include at least one of the following: wrinkle number, wrinkle density, wrinkle length, and wrinkle type.

[0144] Optionally, the target geometry reconstruction parameters include target shape parameters, target pose parameters, and target expression parameters;

[0145] The processor 1001 optimizes the target driving parameters based on wrinkle information and sends the optimized target driving parameters to the rendering display terminal through the communication interface 1003. This allows the rendering display terminal to drive the pre-stored initial parameterized head model according to the optimized target driving parameters, thereby obtaining the target parameterized head model of the target object. The specific operation is as follows:

[0146] Optimize target shape parameters based on wrinkle information;

[0147] Through the communication interface 1003, the target pose parameters, target expression parameters, and optimized target shape parameters are sent to the rendering and display terminal, so that the rendering and display terminal can drive the pre-stored initial parameterized head model to move according to the target pose parameters, target expression parameters, and optimized target shape parameters, thereby obtaining the target parameterized head model of the target object.

[0148] Optionally, the optimization process formula is expressed as follows:

[0149]

[0150] Where base represents the initial driving parameters corresponding to the initial parameterized head model in its natural state. In the natural state, the initial shape and initial expression parameters are 0, and β... i Let ψ represent the target shape parameter of the i-th expression basis under the current expression. i The target expression parameters for the i-th expression base under the current expression, where n is the total number of expression bases.

[0151] Optionally, before sending the optimized target geometry reconstruction parameters to the rendering display terminal via the communication interface 1003, the processor 1001 also performs the following operations:

[0152] Based on the optimized target-driven parameters, set the weight coefficients of each expression base of the target object's face;

[0153] Based on the weight coefficients, the weights of each expression base are adjusted to perform secondary optimization of the target driving parameters.

[0154] It should be noted that, Figure 10 This is merely an example illustrating the hardware necessary for an electronic device to perform the method steps for optimizing face reconstruction parameters provided in the embodiments of this application. Not shown, when the electronic device is... Figure 1 When the data acquisition terminal is used, the electronic device also includes conventional components of VR products such as speakers, microphones, power supplies, and left and right glasses.

[0155] Examples of this application Figure 10 The processor involved can be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0156] This application also provides a computer-readable storage medium for storing instructions that, when executed, can perform the method for optimizing face reconstruction parameters described in the foregoing embodiments.

[0157] This application also provides a computer program product for storing a computer program that executes the method for optimizing face reconstruction parameters in the foregoing embodiments.

[0158] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0160] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for optimizing face-driving parameters, characterized in that, The method, applied to a remote three-dimensional communication system, includes: Before remote 3D interaction, an initial facial image of the target object is acquired, and initial facial key points are extracted from the initial facial image, as well as the gender and age of the target object are identified. Based on the gender and age of the target object, load the corresponding wrinkle information; Based on the wrinkle information, a standard parametric head model is obtained; Based on the initial facial key points and the standard parametric head model, determine the initial parametric head model of the target object; During remote 3D interaction, a target face image of the target object is acquired, and key points of the target face are extracted from the target face image; Based on the initial parameterized head model and the target facial key points, target driving parameters are determined, wherein the target driving parameters include target shape parameters, target pose parameters and target expression parameters; The target shape parameters are optimized based on the wrinkle information, so that the rendering display terminal drives the pre-stored initial parametric head model to move according to the target pose parameters, the target expression parameters, and the optimized target shape parameters, thereby obtaining the target parametric head model of the target object; wherein, the optimization process is expressed by the following formula: Where base represents the initial driving parameters corresponding to the initial parameterized head model in its natural state, where the initial shape parameter and initial expression parameter are 0. This represents the target shape parameter of the i-th expression base under the current expression. The target expression parameters for the i-th expression base under the current expression, where n is the total number of expression bases.

2. The method as described in claim 1, characterized in that, Before loading the corresponding wrinkle information based on the gender and age of the target object, the method further includes: Acquire the voice data of the target object, and identify the gender and age of the target object from the voice data; The gender of the target object identified in the voice data and the initial face image are weighted, and the age of the target object identified in the voice data and the initial face image are weighted, so as to optimize the gender and age of the target object identified in the initial face image.

3. The method as described in claim 1 or 2, characterized in that, The wrinkle information includes at least one of the following: wrinkle number, wrinkle density, wrinkle length, and wrinkle type.

4. The method as described in claim 1, characterized in that, Before sending the optimized target driving parameters to the rendering display terminal, the method further includes: Based on the optimized target driving parameters, set the weight coefficients of each expression base of the target object's face; Based on the weight coefficients, the weights of each expression base are adjusted to perform secondary optimization of the target driving parameters.

5. An electronic device, characterized in that, An application in a remote three-dimensional communication system includes a processor, a memory, and a communication interface, wherein the memory, the communication interface, and the processor are connected via a bus. The memory stores a computer program, and the processor performs the following operations according to the computer program: Before remote 3D interaction, an initial facial image of the target object is acquired, and initial facial key points are extracted from the initial facial image, as well as the gender and age of the target object are identified. Based on the gender and age of the target object, load the corresponding wrinkle information; Based on the wrinkle information, a standard parametric head model is obtained; Based on the initial facial key points and the standard parametric head model, determine the initial parametric head model of the target object; During remote 3D interaction, a target face image of the target object is acquired, and key points of the target face are extracted from the target face image; Based on the initial parameterized head model and the target facial key points, target driving parameters are determined, wherein the target driving parameters include target shape parameters, target pose parameters and target expression parameters; Optimize the target shape parameters based on the wrinkle information; Through the communication interface, the target pose parameters, the target expression parameters, and the optimized target shape parameters are sent to the rendering and display terminal, so that the rendering and display terminal drives the pre-stored initial parametric head model to move according to the target pose parameters, the target expression parameters, and the optimized target shape parameters, thereby obtaining the target parametric head model of the target object. The optimization process is expressed by the following formula: Where base represents the initial driving parameters corresponding to the initial parameterized head model in its natural state, where the initial shape parameter and initial expression parameter are 0. This represents the target shape parameter of the i-th expression base under the current expression. The target expression parameters for the i-th expression base under the current expression, where n is the total number of expression bases.

6. The electronic device as claimed in claim 5, characterized in that, Before the processor loads the corresponding wrinkle information based on the target object's gender and age, it also performs the following operations: Acquire the voice data of the target object, and identify the gender and age of the target object from the voice data; The gender of the target object identified in the voice data and the initial face image are weighted, and the age of the target object identified in the voice data and the initial face image are weighted, so as to optimize the gender and age of the target object identified in the initial face image.

7. The electronic device as claimed in claim 5 or 6, characterized in that, The wrinkle information includes at least one of the following: wrinkle number, wrinkle density, wrinkle length, and wrinkle type.

Images