Virtual image construction method, device, equipment and storage medium

By building a neutral and personalized facial expression base, generating virtual images and transmitting pose parameters and weight coefficients, the lag caused by high-definition facial image transmission in video communication is solved, reducing network bandwidth requirements and protecting privacy.

CN115460372BActive Publication Date: 2024-12-03GUANGZHOU SHIYUAN ELECTRONICS CO LTD +1
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202110645973.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-09
Publication Date
2024-12-03
Estimated Expiration
2041-06-09

AI Technical Summary

Technical Problem

In video communication, transmitting high-definition and real face images leads to high demand for network bandwidth and prone to lag, especially when the network bandwidth is limited.

Method used

By obtaining the face images of the target object collected offline, building a neutral face expression base and a personalized face expression base, obtaining the current face image collected online, determining the positioning parameters and weight coefficients based on these references, and sending them to the remote device To generate virtual images, reduce network bandwidth requirements and protect privacy.

Benefits of technology

It effectively reduces the data processing volume and network bandwidth requirements during video communication, prevents information leakage, ensures the imaging quality of remote devices, and solves the problem of lag.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115460372B_ABST
    Figure CN115460372B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses a virtual image construction method, device, equipment and storage medium, which relates to the field of image processing technology, and includes: obtaining a first image collected offline and including a first face image of a target object; constructing a neutral facial expression base and multiple personalized facial expression bases of the target object based on the first facial image and predetermined expression base parameters; obtaining a second image collected online and including a second facial image of the target object; determining the posture parameters and weight coefficients of each personalized facial expression base required when the three-dimensional face model of the target object is mapped to the second facial image based on the neutral facial expression base and multiple personalized facial expression bases; sending the posture parameters and weight coefficients to a remote device, so that the remote device generates a virtual image corresponding to the second facial image based on the posture parameters and weight coefficients. The above method can solve the technical problem of the jamming phenomenon caused by transmitting real facial images in the related technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a virtual image construction method, device, equipment and storage medium. Background Art

[0002] With the development of network communication technology (especially 5G communication technology), users can use communication equipment (such as mobile phones, computers, etc.) to enjoy network communication resources such as video calls, online classes, and online meetings without leaving home. At present, when using network communication technology for video communication, both parties can see each other's current facial image. In addition, in order to improve the video communication experience of both parties on the call, high-definition images will be transmitted so that both parties on the call can clearly see the corresponding facial images. For example, in an online class or online meeting, the high-definition facial image of the speaker at the main lecture end will be sent to other devices so that users of other devices can view the high-definition facial image. This video communication method has the following defects: the transmission of high-definition images has high requirements for network bandwidth, and when the network bandwidth is limited, it is easy to be stuck. Summary of the invention

[0003] The embodiments of the present application provide a virtual image construction method, device, equipment and storage medium to solve the technical problem of the stuttering phenomenon caused by transmitting real face images in the related art.

[0004] In a first aspect, an embodiment of the present application provides a virtual image construction method, characterized by comprising:

[0005] Acquire a first image acquired offline, where the first image includes a first face image of a target object;

[0006] Constructing a neutral facial expression base and a plurality of personalized facial expression bases of the target object according to the first facial image and predetermined expression base parameters;

[0007] Acquire a second image collected online, where the second image includes a second face image of the target object;

[0008] Determining, based on the neutral facial expression base and the plurality of personalized facial expression bases, the pose parameters required when the three-dimensional facial model of the target object is mapped to the second facial image and the weight coefficients of the personalized facial expression bases;

[0009] The posture parameters and the weight coefficients are sent to a remote device, so that the remote device generates a virtual image corresponding to the second facial image according to the posture parameters and the weight coefficients.

[0010] In a second aspect, the embodiment of the present application further provides a virtual image construction device, including:

[0011] A first image acquisition module, used to acquire a first image acquired offline, wherein the first image includes a first face image of a target object;

[0012] An expression base construction module, used to construct a neutral expression base and multiple personalized expression bases of the face of the target object according to the first face image and predetermined expression base parameters;

[0013] A second image acquisition module, used to acquire a second image collected online, where the second image includes a second face image of the target object;

[0014] A parameter determination module, used to determine the posture parameters required when the three-dimensional face model of the target object is mapped to the second face image and the weight coefficients of each of the personalized face expression bases according to the neutral face expression base and the plurality of personalized face expression bases;

[0015] A parameter sending module is used to send the posture parameters and the weight coefficients to a remote device, so that the remote device generates a virtual image corresponding to the second face image according to the posture parameters and the weight coefficients.

[0016] In a third aspect, the embodiment of the present application further provides a virtual image construction device, including:

[0017] one or more processors;

[0018] A memory for storing one or more programs;

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the virtual image construction method as described in the first aspect.

[0020] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the virtual image construction method as described in the first aspect.

[0021] The above-mentioned virtual image construction method, device, equipment and storage medium obtain a first image containing the first face image of the target object, and construct the face neutral expression base and multiple face personalized expression bases of the target object according to the first image and the predetermined expression base parameters. After that, during the video communication process, obtain the second image collected online containing the second face image of the target object, determine the weight coefficient and posture parameter when the three-dimensional face model constructed according to the face neutral expression base and multiple face personalized expression bases is mapped to the second face image, and then send the posture parameter and weight coefficient to the remote device, so that the remote device displays the virtual image corresponding to the face image through the posture parameter and weight coefficient. The technical means solves the jamming problem caused by the transmission of high-definition real face images in the related technology. Due to the transmission of the weight coefficient of the face personalized expression base and the posture parameter of the three-dimensional face model, the network bandwidth demand is greatly reduced, which is particularly suitable for remote video communication scenarios. Moreover, the target object's neutral facial expression base and personalized facial expression base are determined by the first facial image collected offline. During the video communication process, the neutral facial expression base and personalized facial expression base can be used directly without reconstructing the neutral facial expression base and personalized facial expression base, which reduces the amount of data processing during the video communication process. In addition, the transmitted weight coefficient and posture parameter can enable the remote device to display the corresponding virtual image, effectively protecting the privacy of the target object and preventing information leakage. At the same time, the virtual image obtained according to the weight coefficient and posture parameter accurately follows the expression and posture in the second facial image, effectively ensuring the imaging quality of the remote device. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of a virtual image construction method provided in one embodiment of the present application;

[0023] Figure 2 A schematic diagram of a second image provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a virtual image provided in an embodiment of the present application;

[0025] Figure 4 A flowchart of a virtual image construction method provided in an embodiment of the present application;

[0026] Figure 5 A flowchart of another virtual image construction method provided by an embodiment of the present application;

[0027] Figure 6 A schematic diagram of expression transmission provided in an embodiment of the present application;

[0028] Figure 7 A first schematic diagram of key points of a face provided in an embodiment of the present application;

[0029] Figure 8 A second schematic diagram of key points of a face provided in an embodiment of the present application;

[0030] Fig. 9 A third schematic diagram of key points of a face provided in an embodiment of the present application;

[0031] Fig.10 A schematic diagram of a reference three-dimensional face model provided in an embodiment of the present application;

[0032] Fig.11 A schematic diagram of a second facial image provided in an embodiment of the present application;

[0033] Fig.12 A schematic diagram of a three-dimensional face model of a target object provided in an embodiment of the present application;

[0034] Fig.13 A flowchart of another virtual image construction method provided for an embodiment of the present application;

[0035] Fig.14 A schematic diagram of mutually exclusive expressions provided in an embodiment of the present application;

[0036] Fig.15 Another mutually exclusive expression basic schematic diagram provided in the embodiment of the present application;

[0037] Fig.16 Another schematic diagram of mutually exclusive expressions provided in an embodiment of the present application;

[0038] Fig.17 A fourth schematic diagram of key points of a face provided in an embodiment of the present application;

[0039] Fig.18 A schematic diagram of the structure of a virtual image construction device provided in an embodiment of the present application;

[0040] Fig.19 A schematic diagram of the structure of a virtual image construction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0042] The virtual image construction method provided in the embodiment of the present application can be executed by a virtual image construction device, which can be implemented by software and / or hardware, and the virtual image construction device can be composed of two or more physical entities, or can be composed of one physical entity. For example, the virtual image construction device can be a computer, a mobile phone, a tablet computer, or an interactive smart tablet or other smart device.

[0043] In the embodiment, the virtual image construction device is applied in scenarios such as online conferences and online classes that use network communication technology for video communication. In this scenario, in addition to the virtual image construction device, other devices participating in the video communication are also included. The other devices can be one or more, and the other devices can also be smart devices such as computers, mobile phones, tablet computers or interactive smart tablets. During video communication, the virtual image construction device executes the virtual image construction method provided in this embodiment to process when collecting the facial image of the local user, so that other devices can display the virtual image obtained based on the facial image. At this time, the other device is a remote device relative to the virtual image construction device. It can be understood that in actual applications, the virtual image construction method provided in this embodiment can also be executed when the remote device collects the facial image of the user. At this time, the remote device can also be considered as a virtual image construction device, and the local device is used to display the corresponding virtual image. For example, in the online classroom scenario, the device used by the lecturer can be considered as a virtual image acquisition device, and the device used by the student can be considered as a remote device. For another example, in the online conference scenario, the device used by the current speaker can be considered as a virtual image construction device, and the devices used by other participants can be considered as remote devices.

[0044] In the embodiment, the virtual image construction device is installed with at least one type of operating system, wherein the operating system includes but is not limited to Android system, IOS system and / or Windows system, etc. The virtual image construction device can install at least one application based on the operating system, and the application can be an application that comes with the operating system, or an application downloaded from a third-party device or server. The category of the application is not limited in the embodiment. It can be understood that the virtual image construction method provided in the embodiment of the present application can also be the application itself. In the embodiment, the virtual image construction device is at least installed with an application for executing the virtual avatar construction method provided in the embodiment of the present application, and the virtual avatar construction method is executed when the application is running. In the embodiment, the virtual image construction device can also be installed with an image acquisition device, which can be a camera, and image acquisition can be performed through the image acquisition device. For example, in an online classroom scenario, the virtual image acquisition device can collect continuous frame images of the lecturer during the teaching process, so as to obtain the video data of the lecturer during the teaching process through the continuous frame images, thereby realizing video communication. In an embodiment, the virtual image construction device may also be installed with an audio acquisition device, which may be a microphone. Audio acquisition may be performed through the audio acquisition device. For example, in an online classroom scenario, the virtual image acquisition device may collect the audio data of the lecturer during the lecture and send it to a remote device so that the remote device plays the audio data.

[0045] Figure 1 A flowchart of a virtual image construction method provided by an embodiment of the present application. Figure 1 , the virtual image construction method specifically includes:

[0046] Step 110: Acquire a first image collected offline, where the first image includes a first face image of the target object.

[0047] The first image is a frame of image captured offline. Offline capture refers to capture performed before video communication. At this time, the first image can be considered as an image captured in the offline stage (before video communication or network communication), and the captured first image will not be transmitted to the remote device. In one embodiment, the first image includes a facial image of the target object. Among them, the target object refers to the object for which a virtual image needs to be generated. Any object that can be identified as a facial image can be considered as the target object without prior designation. For example, in an online classroom scenario, the target object can be a lecturer who uses a virtual image to build a device. At this time, the facial image of the target object refers to the facial image of the lecturer. In the embodiment, the facial image of the target object in the first image is recorded as the first facial image. It can be understood that the embodiment of the expression of the target object in the first facial image is not limited.

[0048] The number of target objects is one or more, and in the embodiment, one target object is used for exemplary description. In practical applications, when there are multiple target objects, the processing method for each target object is the same as the processing method for the current target object.

[0049] Optionally, before video communication, an image acquisition device is used to capture a frame of image, and then it is confirmed whether the image contains the face image of the target object. If so, it is determined that the first image containing the first face image is captured, and the subsequent steps are executed. Otherwise, this step is repeated. The technical means used to determine whether the face image is contained are not limited in the embodiment. For example, a face detection algorithm based on deep learning is used to detect the face image area in the captured image. If the face image area is detected, it is determined that the face image is contained. Otherwise, it is determined that the face image is not contained.

[0050] It should be noted that this step can be considered as a step of pre-shooting the face of the target object before starting video communication. Through this step, a first image including a first face image can be obtained.

[0051] Step 120: construct a neutral facial expression base and multiple personalized facial expression bases of the target object according to the first facial image and predetermined expression base parameters.

[0052] The expression base can be understood as a three-dimensional face model containing the position information of the key points of the face. The expression of a person can be reflected through the position information of the key points of the face in the expression base. The key points of the face can be obtained through the key parts of the face, where the key parts of the face include eyebrows, eyes, nose, mouth and cheeks, etc. The key points of the face refer to the points located in the above key parts and used to describe the current action of the key parts. At this time, the action of each key part can be determined through the key points of the face, and then the face posture, face position and face expression can be determined, that is, each key point of the face is the semantic information of the face.

[0053] In one embodiment, facial expressions are divided into neutral expressions and personalized expressions. Neutral expressions refer to the morphology of a face without any expression, which can reflect the identity of the face, wherein the identity of the face is a specific description of the morphology of the face, such as the face identity describes the key parts of the face, for example, the key parts described by the face identity are big eyes, high nose bridge, and thin lips. At this time, since the faces of different objects are different, there will be differences between the key parts described by the face identities of different objects. Personalized expressions refer to the expressions made by the face, such as closing eyes, opening mouth, frowning, etc. Correspondingly, the expression base is divided into a neutral expression base and a personalized expression base according to the facial expression. Among them, the neutral expression base can be understood as an expression base representing a neutral expression, and the morphology of the face in three-dimensional space when there is no expression can be confirmed through the neutral expression base. The personalized expression base refers to an expression base containing personalized expressions, and each personalized expression base corresponds to a personalized expression. It can be understood that since the expressions of the face are very rich, if you want to express all the expressions of the face, you need to build a large number of personalized expression bases for the face, which will greatly increase the amount of data processing. Therefore, in the embodiment, only a personalized expression base of basic expressions is constructed, wherein the specific content of the basic expressions can be set according to the actual situation, and various expressions of the human face can be obtained by combining the basic expressions and the neutral expressions. For example, the basic expressions for the eyes include: left eye closed, left eye wide open, right eye closed and right eye wide open. At this time, various expressions of the eyes can be obtained according to the above four basic expressions and the neutral expression. For example, the expression of squinting eyes can be obtained by linearly superimposing the left eye closed, the right eye closed and the neutral expression. In the embodiment, the neutral expression base representing the neutral expression of the target object is recorded as the neutral expression base of the human face, and the personalized expression base representing the personalized expression of the target object is recorded as the personalized expression base of the human face.

[0054] The neutral facial expression base of the target object can be constructed through the first facial image. In one embodiment, prior information is introduced when constructing the neutral facial expression base. The prior information is obtained by collecting a large amount of three-dimensional facial data, which can reflect the average coordinate data, facial identity basis vectors and personalized expression basis vectors of a large amount of three-dimensional facial data. A three-dimensional facial model can be constructed through the prior information. The three-dimensional facial model can be considered as a reference three-dimensional facial model constructed based on the prior information. It can be understood that there are differences in the reference three-dimensional facial model obtained when different coefficients are set for the prior information. By setting different coefficients to adjust the reference three-dimensional facial model, a three-dimensional facial model corresponding to the first facial image can be obtained. In one embodiment, the coordinates of each facial key point in the first facial image in a two-dimensional plane are first obtained, and then the facial key points of the reference three-dimensional facial model (the facial key points are three-dimensional key points) are projected into the two-dimensional plane to determine the coordinates of the three-dimensional key points in the two-dimensional plane. Then, the error between the facial key points in the reference three-dimensional facial model and the facial key points in the first facial image after being projected into the two-dimensional plane is calculated. It can be understood that the facial key points for which the error is to be calculated have a corresponding relationship, that is, the corresponding facial key points in the reference three-dimensional facial model and the first facial image belong to the same relative position of the same key part, for example, the left boundary point of the left eye in the reference three-dimensional facial model and the first facial image are facial key points with a corresponding relationship. Then, the position of the facial key points in the reference three-dimensional facial model is adjusted according to the calculated error to ensure that the adjusted facial key points of the reference three-dimensional facial model coincide with the coordinates of the facial key points in the facial image as much as possible when projected into the two-dimensional plane. Among them, adjusting the position of the three-dimensional key point can be achieved by adjusting the coefficient used in the prior information. When the coefficient is different, the position of the three-dimensional key point in the reference three-dimensional face model constructed by the prior information is different. When the facial key points overlap as much as possible, it can be considered that the adjusted reference three-dimensional face model is close to the three-dimensional face model of the target object. After that, the coefficient used in the end is obtained, and the personalized expression basis vector in the prior information is removed, so as to construct a reference three-dimensional face model that does not contain personalized expressions through the coefficient used in the end and the prior information after removing the personalized expression basis vector. At this time, the reference three-dimensional face model can be considered as the neutral facial expression base of the target object. It should be noted that in actual applications, other methods can also be used to construct the neutral facial expression base, such as using a neural network method, and the first facial image or the facial key points in the first facial image are input into the neural network to obtain the corresponding neutral facial expression base.

[0055] Afterwards, a personalized facial expression base of the target object is constructed according to the neutral facial expression base and the expression base parameters. Each personalized facial expression base corresponds to an expression base parameter. The expression base parameter can reflect the difference between the corresponding personalized expression base and the neutral expression base. By adding the difference to the neutral facial expression base, the corresponding personalized facial expression base can be obtained. In one embodiment, the expression base parameter can be a predetermined parameter so that the expression base parameter is directly used in this step. The embodiment of the determination method of the expression base parameter is not limited. For example, the average coordinate data of a large amount of three-dimensional facial data (the same as the three-dimensional facial data collected when determining the neutral facial expression base) is determined as a neutral expression base. In the embodiment, the neutral expression base is recorded as the benchmark neutral identity information. Afterwards, each personalized expression base of the benchmark neutral identity information is constructed. In the embodiment, the personalized expression base corresponding to the benchmark neutral identity information is recorded as the benchmark personalized expression base. Afterwards, the difference between the benchmark personalized expression base and the benchmark neutral identity information is determined, and the difference is used as the expression base parameter. The corresponding expression base parameters can be obtained by calculating the coordinate difference between the reference personalized expression base and the reference neutral identity information in three-dimensional space.

[0056] It should be noted that the expression bases involved in the embodiment are located in the same coordinate system in the three-dimensional space, and the facial images involved are located in the same coordinate system in the two-dimensional space.

[0057] It can be understood that for the same target object, before each video communication, the target object's neutral facial expression base and personalized facial expression base can be obtained according to the above steps. Alternatively, the target object's neutral facial expression base and personalized facial expression base are pre-constructed once, and before each video communication, the pre-constructed neutral facial expression base and personalized facial expression base are directly obtained. Optionally, the first facial image of the target object can be collected at a fixed time to update the pre-constructed neutral facial expression base and personalized facial expression base.

[0058] Step 130: Acquire a second image collected online, where the second image includes a second facial image of the target object.

[0059] The second image is the current frame image collected during the video communication process, that is, the second image can be understood as a frame image collected during the online process. It can be understood that during the video communication process, the virtual image construction device will continuously collect multiple frames of second images, and each frame of the second image is processed in the same way. In the embodiment, a frame of the second image is used as an example for description.

[0060] In one embodiment, the facial image of the target object in the second image is recorded as the second facial image. Optionally, after acquiring the second image, first confirm whether the second image contains the facial image of the target object. If so, determine that the second facial image is acquired and execute subsequent steps. Otherwise, stop executing subsequent steps and acquire the next frame of image as the currently acquired second image, and repeat this step.

[0061] Step 140: Determine, based on the neutral facial expression base and the multiple personalized facial expression bases, the posture parameters required when the three-dimensional facial model of the target object is mapped to the second facial image and the weight coefficients of the personalized facial expression bases.

[0062] Exemplarily, a weight coefficient is set for each personalized facial expression base, and then each personalized facial expression base is linearly weighted with the neutral facial expression base in combination with the weight coefficient, so that a three-dimensional facial model of the target object with expression can be obtained. It should be noted that since the specific expression of the second facial image cannot be clearly defined at present, it is impossible to set an accurate weight coefficient for each personalized facial expression base. Therefore, when constructing the three-dimensional facial model, an initial weight coefficient is first set for each personalized facial expression base. At this time, there may be a difference between the expression represented by the constructed three-dimensional facial model and the real expression of the second facial image.

[0063] It can be understood that the closer the expression represented by the three-dimensional face model is to the real expression of the second face image, the smaller the difference between the three-dimensional face model and the second face image after the three-dimensional face model is mapped to the two-dimensional plane. Therefore, in the embodiment, the weight coefficients of the personalized expression bases of each face when constructing the three-dimensional face model can be continuously adjusted so that the expression represented by the three-dimensional face model is close to the real expression of the second face image. At the same time, since the head of the target object in the second image will have certain movements, such as tilting the head, turning the head, etc., therefore, in the embodiment, it is also necessary to adjust the posture parameters of the three-dimensional face model so that the movement of the adjusted three-dimensional face model is consistent with the head movement of the target object in the second face image. Among them, the posture parameters can also be understood as rigid transformation parameters. In the embodiment, rigid transformation refers to changing the position, orientation and size of the three-dimensional face model without changing the shape. Rigid transformation parameters refer to parameters used when rigidly transforming the three-dimensional face model. In the embodiment, rigid transformation parameters include: rigid rotation matrix, translation vector and scale scaling factor. The rigid rotation matrix is ​​used to change the orientation of the three-dimensional face model, the translation vector is used to change the position of the three-dimensional face model, and the scale factor is used to change the size of the three-dimensional face model. It can be understood that by continuously adjusting the position parameters of the three-dimensional face model and the weight coefficients of each personalized facial expression base, the posture parameters required when the three-dimensional face model is mapped to the second face image (that is, the image obtained when the three-dimensional face model is mapped to the two-dimensional plane is infinitely close to the second face image) and the weight coefficients of each personalized facial expression base can be obtained.

[0064] In one embodiment, the difference between the two-dimensional image obtained when the three-dimensional face model is mapped to the two-dimensional plane and the second face image is represented by constructing an error parameter formula. Among them, the smaller the error parameter, the smaller the difference, the closer the expression represented by the three-dimensional face model is to the real expression of the second face image, and the more consistent the action of the three-dimensional face model is with the head action of the target object in the second face image. In an optional manner, the error parameter formula is constructed by the coordinate difference between the two-dimensional image when the three-dimensional face model is mapped to the two-dimensional plane and the face key points in the second face image. It can be understood that the coordinates of the face key points corresponding to the three-dimensional face model can be determined by the weight coefficient and the posture parameter. Therefore, when constructing the error parameter formula, the weight coefficient and the posture parameter can be considered as unknown quantities. In the subsequent processing process, the weight coefficient and the posture parameter can be continuously adjusted to make the coordinates of the face key points corresponding to the three-dimensional face model in the two-dimensional image and the coordinates of the face key points in the second face image closer and closer, thereby making the error parameter smaller and smaller, the expression represented by the three-dimensional face model and the real expression of the face image are more and more the same, and the action of the three-dimensional face model and the posture parameters of the head action of the target object in the face image are more and more consistent. Exemplarily, when the calculated error parameter has reached the expected value, the currently used weight coefficient and posture parameter can be used as the final weight coefficient and posture parameter. Among them, it is possible to determine whether the error parameter has reached the expected value by setting the number of adjustments of the weight coefficient and posture parameter, that is, when the number of adjustments reaches a certain number, it is determined that the error parameter has reached the expected value. It is also possible to determine whether the error parameter has reached the expected value by setting a parameter threshold, that is, when the error parameter is lower than the parameter threshold, it is determined that the error parameter has reached the expected value. Generally speaking, if the error parameter has reached the expected value, it can be considered that the expression represented by the three-dimensional face model is sufficiently similar to the real expression of the second face image, and the movement of the three-dimensional face model is sufficiently consistent with the posture parameters of the head movement of the target object in the second face image.

[0065] Step 150: Send the posture parameters and weight coefficients to the remote device, so that the remote device generates a virtual image corresponding to the second face image according to the posture parameters and weight coefficients.

[0066] Exemplarily, the posture parameters and weight coefficients are sent to a remote device that communicates with the virtual image construction device via video. A virtual image is stored in the remote device, which may be a cartoon image, a two-dimensional virtual image, or a three-dimensional virtual image. In the embodiment, a three-dimensional virtual image is taken as an example. The storage of the three-dimensional virtual image in the remote device specifically includes storing a neutral expression base and a personalized expression base of the three-dimensional virtual image, wherein each personalized expression base of the three-dimensional virtual image has the same expression as the corresponding personalized expression base of the human face. In one embodiment, a user of the remote device can install an application in the remote device, which can receive the posture parameters and weight coefficients sent by the virtual image construction device, and process the virtual image according to the posture parameters and weight coefficients to generate a virtual image with the same expression and the same head movement as the second face image, and the remote device stores the virtual image when the application is installed. When the application is upgraded or updated, the virtual image stored in the remote device can be updated at the same time.

[0067] In one embodiment, when the remote device generates a virtual image corresponding to the second facial image, the pre-set three-dimensional virtual image may be rendered and displayed through the graphics rendering framework of the Open Graphics Library (OpenGL). Specifically, during rendering, the personalized expression base and the neutral expression base of the three-dimensional virtual image are linearly weighted according to the weight coefficient to obtain a three-dimensional virtual image containing an expression, wherein the linear weighting method is the same as the linear weighting method when constructing the three-dimensional face model in step 140. After generating the three-dimensional virtual image containing the expression, the graphics rendering framework performs a corresponding rigid transformation on the three-dimensional virtual image containing the expression according to the posture parameters, and displays it after the rigid transformation is completed. At this time, the displayed three-dimensional virtual image not only has the same expression as the expression in the facial image, but also has the same posture as the posture in the facial image. For example, Figure 2 A schematic diagram of a second image provided in an embodiment of the present application. Figure 3 A schematic diagram of a virtual image provided in an embodiment of the present application. Figure 2 After the second image shown is processed, the remote device can display Figure 3 The virtual image schematic diagram shown is as follows: Figure 2 The mouth of the face image is open, and the remote device displays Figure 3 The mouth of the virtual image is also open.

[0068] It can be understood that after sending the weight coefficient and posture parameter to the remote device, the next frame image can be obtained, and the next frame image can be used as the second image collected online, and the above process is repeated so that the remote device display continuously displays the virtual image.

[0069] Optionally, a function control for opening a virtual image can also be set in the virtual image construction device. The function control can be implemented by a physical button or a virtual button. When the function control is detected to be triggered, the weight coefficient and posture parameter corresponding to the second face image are determined and sent to the remote device so that the remote device displays the virtual image. When the function control is detected to stop being triggered, only the currently captured second image needs to be sent to the remote device so that the remote device displays the second image. In this way, the user can determine whether to display the real image according to his own needs, thereby improving the user experience.

[0070] In the above, by acquiring a first image including a first face image of a target object, and constructing a neutral face expression base and multiple personalized face expression bases of the target object based on the first image and predetermined expression base parameters, then, during the video communication process, acquiring a second image currently collected including a second face image of the target object, determining the weight coefficient and posture parameter when the three-dimensional face model constructed based on the neutral face expression base and multiple personalized face expression bases is mapped to the second face image, and then, sending the posture parameter and weight coefficient to a remote device so that the remote device displays a virtual image corresponding to the face image through the posture parameter and weight coefficient. This technical means solves the problem of stuttering caused by transmitting high-definition real face images in related technologies. Due to the transmission of the weight coefficient of the personalized face expression base and the posture parameter of the three-dimensional face model, the demand for network bandwidth is greatly reduced, which is particularly suitable for remote video communication scenarios. Moreover, the target object's neutral facial expression base and personalized facial expression base are determined by the first facial image collected offline. During the video communication process, the neutral facial expression base and personalized facial expression base can be used directly without rebuilding the neutral facial expression base and personalized facial expression base, which reduces the amount of data processing during the video communication process. In addition, the transmitted weight coefficient and posture parameters can enable the remote device to display the corresponding virtual image, effectively protecting the privacy of the target object and preventing information leakage. At the same time, the virtual image accurately follows the expression and posture in the facial image, ensuring the imaging quality of the remote device.

[0071] The following description takes an online class as an example. At this time, the virtual image construction device is the lecturer (i.e., the device used by the teacher), and the remote device is the client (i.e., the device used by the student). Based on the above virtual image construction method, it can be seen that the virtual image construction process can be divided into three stages: Figure 4 For a flowchart of a virtual image construction method provided in an embodiment of the present application, refer to Figure 4, the three stages are pre-offline stage, offline stage and online stage. The pre-offline stage is used to construct a universal expression base parameter, which can be used as prior information. The offline stage is used to determine the teacher's facial neutral expression base and facial personalized expression base through the teacher's facial image and prior information. This process only needs to be performed once before an online class starts, so the tolerance for speed is high. The online stage is used to estimate the weight coefficients of each facial personalized expression base and the pose parameters of the three-dimensional face model according to the teacher's real-time second image during the online class. Afterwards, the weight coefficients and pose parameters are sent to the client so that the client can render and generate the corresponding virtual image.

[0072] Figure 5 A flowchart of another virtual image construction method provided in one embodiment of the present application. This embodiment is specific based on the above embodiment. Figure 5 , the virtual image construction method specifically includes:

[0073] Step 210: Obtain pre-constructed baseline neutral identity information, a reference neutral expression base, and a plurality of reference personalized expression bases.

[0074] In one embodiment, the benchmark neutral identity information is determined by the three-dimensional face data in the publicly available BFM (Basel Face Mode) database. Each three-dimensional face data can be considered as a three-dimensional face model, and the expression embodiment of the three-dimensional face model is not limited. Exemplarily, the average coordinate data of the three-dimensional face data in the BFM database is determined as the benchmark neutral identity information. Optionally, the average coordinate data of 200 three-dimensional face data in the BFM database is determined as the benchmark neutral identity information. In one embodiment, the benchmark neutral identity information is recorded as MU, and MU has a total of 3h data, where h refers to the average number of point clouds of 200 three-dimensional face data, and each point cloud contains coordinates of the three axes of x, y, and z. A three-dimensional face can be constructed through MU. In the embodiment, the average coordinate data represented by MU can be understood as a neutral expression base representing an average face.

[0075] The reference neutral expression base is a pre-set expression base representing a neutral expression. The reference personalized expression base is an expression base obtained by adding a pre-set basic expression to the reference neutral expression base. Each reference personalized expression base has a corresponding physical meaning. In one embodiment, the facial action coding system (FACS) is used to define each muscle action of the face as a different action unit AU value or AD value, that is, each basic expression is classified by muscle action. For example, the AU value corresponding to "pull the inner eyebrow upward" is recorded as AU1. In addition, each AU value also includes a refinement value, which is used to indicate the movement amplitude of the muscle. For example, if the AU value including the refinement value is AU1 (0.2), it means that the current basic expression is the inner eyebrow pulling upward, and the pulling degree is 0.2. For another example, the AU value corresponding to "eyes closed" is recorded as AU43, then AU43 (0) indicates that the eyes are normally open, and AU43 (1) indicates that the eyes are completely closed. At this time, the refinement value is between 0 and 1, which indicates different degrees of eye opening.

[0076] In one embodiment, 26 basic expressions are defined according to muscle movements, and each basic expression corresponds to a reference personalized expression base. At this time, each reference personalized expression base, the corresponding basic expression and the AU value are shown in the following table:

[0077]

[0078]

[0079] Table 1

[0080] Among them, Blendshape represents a personalized expression base, 0-25 is the number of the 26 personalized expression bases, the custom expression is the basic expression corresponding to each expression base, and the FACS definition represents the AU value or AD value corresponding to each personalized expression base. It can be seen from the above custom expressions that each personalized expression base mainly divides some left-right symmetrical expressions, so that when the target object has an asymmetrical expression, the corresponding face personalized expression base can be accurately constructed. According to the above table, after adding 26 expressions to the reference neutral expression base, 26 reference personalized expression bases can be obtained. It can be understood that in actual applications, other personalized expression bases can also be constructed, and the embodiment does not limit this.

[0081] Step 220: Determine deformation information between the reference neutral expression base and each reference personalized expression base.

[0082] The deformation information is also considered as the transfer deformation amount from the reference neutral expression base to the reference personalized expression base. That is, after processing the reference neutral expression base using the deformation information, the reference personalized expression base can be obtained. It can be understood that each reference personalized expression base corresponds to one deformation information.

[0083] In one embodiment, the deformation information of the reference neutral expression base and the reference personalized expression base is determined by using a three-dimensional network deformation transfer method. Exemplarily, after the facial key points in the reference neutral expression base are triangulated in the order of arrangement using the Delaunay triangulation algorithm, the reference neutral expression base can be divided into a plurality of triangular facets, and the three vertices of each triangular facet are three facial key points that form a triangle, and each triangular facet can form a three-dimensional mesh representing the reference neutral expression base. Similarly, after the facial key points in the reference personalized expression base are triangulated in the order of arrangement using the Delaunay triangulation algorithm, the reference personalized expression base can be divided into a plurality of triangular facets, and the three vertices of each triangular facet are three facial key points that form a triangle, and each triangular facet can form a three-dimensional mesh representing the reference personalized expression base.

[0084] Each triangle in the reference personalized expression base corresponds to each triangle in the reference neutral expression base one by one. According to the corresponding relationship, the deformation information of the triangle in the reference neutral expression base to the corresponding triangle in the reference personalized expression base can be determined. Among them, the deformation information represents the transfer deformation variable (rotation matrix, translation vector, scale scaling factor, etc.) used when the triangle in the reference neutral expression base is deformed, so that the deformed triangle is the same as the corresponding triangle in the reference personalized expression base. Each triangle corresponds to a deformation information. The deformation information of each triangle constitutes the deformation information of the reference neutral expression base to the current reference personalized expression base.

[0085] Step 230: Determine a plurality of reference personalized expression bases relative to the reference neutral identity information according to the deformation information and the reference neutral identity information, wherein each reference personalized expression base corresponds to a reference personalized expression base.

[0086] Optionally, the baseline neutral identity information and the reference neutral expression base are subjected to three-dimensional network alignment. In one embodiment, an iterative closest point (ICP) algorithm is used to implement three-dimensional mesh alignment. Thereafter, the correspondence between each triangular facet in the aligned reference neutral expression base and each triangular facet in the baseline neutral identity information is determined by a kd tree, wherein the kd tree can be understood as a data structure for organizing points in a k-dimensional Euclidean space.

[0087] Afterwards, according to the correspondence between each triangular facet in the reference neutral expression base and each triangular facet in the reference neutral identity information, the corresponding triangular facet in the reference neutral identity information of the person is processed using the deformation information of each triangular facet when the reference neutral expression base is transformed into a certain reference personalized expression base, that is, the triangular facet is deformed so that the deformed triangular facet is used as the triangular facet in the corresponding personalized expression base. In this way, after processing each triangular facet in the reference neutral identity information, it is possible to clearly obtain a personalized expression base after adding the basic expression corresponding to the reference personalized expression base to the reference neutral identity information. In the embodiment, the obtained personalized expression base is recorded as the reference personalized expression base. After processing in the above manner, each reference personalized expression base has a corresponding reference personalized expression base.

[0088] For example, Figure 6 This is a schematic diagram of expression transmission provided by the embodiment of the present application. Figure 6 The first column of the first row is the reference neutral expression base, and the second to fourth columns of the first row are three reference personalized expression bases. Based on the reader's vision, the corresponding basic expressions are closing the right eye, opening the mouth, and pouting. The first column of the second row is the reference neutral identity information (i.e., MU). The deformation information is determined based on the reference neutral expression base and each reference personalized expression base. After that, the neutral expression base of the face is processed according to the deformation information to obtain the reference personalized expression base. Figure 6 The second row, second column to fourth column are benchmark personalized expression bases obtained according to the reference personalized expression bases in the first row, second column to fourth column. At this time, the basic expressions corresponding to the three benchmark personalized expression bases are closing the right eye, opening the mouth and pouting the mouth, respectively, which realizes the transfer of the basic expressions from the reference personalized expression base to the benchmark personalized expression base.

[0089] Step 240: Determine expression base parameters between the reference personalized expression base and the reference neutral identity information, where each reference personalized expression base corresponds to one expression base parameter.

[0090] In the embodiment, the difference between the reference neutral identity information and each reference personalized expression base in the three-dimensional space is determined, and the difference is determined as the expression base parameter corresponding to the reference personalized expression base. Based on this, the expression base parameter can be obtained using the following formula:

[0091] ΔB i =B i -MU

[0092] Among them, MU represents the baseline neutral identity information, B i represents the i-th benchmark personalized expression base, ΔB i Indicates B i The corresponding expression base parameter is 1≤i≤n, where n is the total number of benchmark personalized expression bases.

[0093] According to the above formula, the expression base parameters corresponding to each benchmark personalized expression can be obtained. It can be understood that by adding the expression base parameters on the basis of the benchmark neutral identity information, the corresponding benchmark personalized expression base can be obtained.

[0094] In one embodiment, after the expression base parameters are obtained, the expression base parameters are saved for subsequent direct use. It is understandable that when the target object changes, there is no need to recalculate the expression base parameters, and the expression base parameters can be directly used.

[0095] Step 250: Acquire a first image collected offline, where the first image includes a first face image of the target object.

[0096] Step 260: Determine a first coordinate array of each facial key point in the first facial image.

[0097] In one embodiment, a face recognition algorithm is used to detect the location area of ​​the target object's face in the first image, and then the location area of ​​the face is extracted to obtain a first face image. Then, using the face key point detection technology, face key points are detected in the first face image, and the coordinates (two-dimensional coordinates) of the detected face key points in the first face image are obtained, and the coordinates of each face key point are combined into a key point coordinate array. In the embodiment, the key point coordinate array obtained based on the first face image is recorded as the first coordinate array.

[0098] In one embodiment, the target object's face has 68 key points. Figure 7 The first schematic diagram of key points of a face provided in an embodiment of the present application. Figure 7 , based on the reader's vision, it shows the facial key points contained in the left eyebrow and the left eye in the current face image, wherein the left eyebrow contains 5 key points and the left eye contains 6 key points. It can be understood that the right eyebrow and the right eye, which are not shown, also contain 5 key points and 6 key points respectively. Figure 8 The second schematic diagram of key points of a face provided in an embodiment of the present application. Figure 8 , which shows that the mouth in the current face image contains 20 key points and the nose contains 9 key points. Fig. 9 The third schematic diagram of key points of a face provided in an embodiment of the present application. Fig. 9 , which shows that the cheek in the current face image contains 17 key points. Each of the above-mentioned face key points has corresponding face semantic information.

[0099] It can be understood that after the facial key point detection is performed on the first facial image, the coordinates of 68 facial key points in the first facial image are obtained, and the coordinates are arranged in a certain order to form a first coordinate array. Among them, the arrangement order of the facial key point coordinates can be set according to actual conditions, and the embodiment does not limit it.

[0100] Step 270: construct a neutral facial expression base of the target object according to the first coordinate array, the first facial image and the reference neutral identity information.

[0101] In the embodiment, the prior information when constructing the neutral facial expression base is based on the BFM database corresponding to the reference neutral identity information. At this time, the BFM database corresponding to the reference neutral identity information is used when determining the neutral facial expression base and the expression base parameters. In one embodiment, the principal component analysis method (Principal Component Analysis, PCA) is used to extract 200 three-dimensional face data from the BFM database to obtain a bilinear model, wherein the bilinear model is a reference three-dimensional face model constructed based on the 200 three-dimensional face data, and its specific model expression formula is:

[0102] M=MU+PC id α id +PC exp α exp

[0103] Among them, M is the reference 3D face model, MU is the benchmark neutral identity information, and MU can be used to construct a 3D face. id The face identity basis vector is obtained through 200 three-dimensional face data, through PC id It can be reflected as the face identity superimposed on MU, that is, by superimposing PC on MU id The face identity of the reference 3D face model (e.g., the neutral expression face features when there is no expression) can be obtained. exp The personalized expression basis vectors obtained from 200 three-dimensional face data are obtained through PC exp It can be reflected as a personalized expression superimposed on MU, that is, by superimposing PC on MU exp The personalized expression of the reference 3D face model can be obtained, α id is the coefficient corresponding to the face identity basis vector, α exp is the coefficient corresponding to the personalized expression basis vector. id and α exp PC id and PC exp By performing linear weighting and integrating the weighted result into the average coordinate data of the 3D face data, a reference 3D face model can be obtained. It can be understood that MU, PC id 、PCexp After constructing the reference 3D face model, the reference 3D face model can be mapped to a 2D plane to determine the difference between the 2D image in the 2D plane and the first face image, and then the prior information PC can be adjusted according to the difference. id and PC exp Alpha used id and α exp , so that the reference three-dimensional face model obtained after adjustment is mapped to a two-dimensional plane and is highly similar or identical to the first face image.

[0104] In one embodiment, an energy constraint formula between the reference 3D face model and the first face image is constructed, and then α is determined by the energy constraint formula. id and α exp , so that the reference 3D face model is highly similar or identical to the first face image after being mapped to the 2D plane. The energy constraint formula can be:

[0105]

[0106] Among them, E lan (p) represents the energy constraint between the reference 3D face model and the first face image, p represents the parameters used by the reference 3D face model, and p includes the coefficient α corresponding to the face identity basis vector id , coefficient α corresponding to the personalized expression basis vector exp , weak perspective projection matrix ∏ and rigid transformation matrix φ, where weak perspective projection is mainly used to project three-dimensional space point information (such as a reference three-dimensional face model) to a two-dimensional imaging plane (such as the two-dimensional plane where the first face image is located), the weak perspective projection matrix refers to the matrix used when projecting the reference three-dimensional face model to a two-dimensional plane, and the rigid transformation matrix may include a rigid rotation matrix, a translation vector, and a scale scaling factor. ω conf,j represents the confidence of the j-th facial key point detection in the first face image, f j represents the coordinates of the jth facial key point in the first face image, F represents the first coordinate array, v j represents the coordinates of the jth three-dimensional key point when the reference three-dimensional face model is mapped to a two-dimensional plane. It can be understood that the arrangement order of the facial key points when the reference three-dimensional face model is mapped to a two-dimensional plane is the same as the arrangement order of the facial key points in the first face image. In one embodiment, when the reference three-dimensional face model is mapped to a two-dimensional plane, the more similar the coordinates of each facial key point are to the corresponding facial key point in the face image, the higher the value of E lan The smaller (p) is, the closer the reference 3D face model is to the 3D face model of the target object in the first face image. Therefore, in the embodiment, the parameters of the reference 3D face model are continuously adjusted so that E lan(p) becomes smaller and smaller (that is, the projection error of the facial key points becomes smaller and smaller). lan (p) After stabilization, determine E lan The specific value of p when (p) is the smallest, at which point the projection error between the facial key points in the reference 3D face model and the facial key points in the first face image is the smallest. lan When (p) is minimum, remove α exp , ∏ and φ, take α id Specific value of and substitute it into M=MU+PC id α id The final reference 3D face model is obtained. At this time, the obtained reference 3D face model is closest to the 3D face model of the target object without any expression. Therefore, the obtained reference 3D face model can be determined as the neutral expression base of the face of the target object. Therefore, this embodiment reconstructs the face identity information of the target object only through the single frame first face image.

[0107] For example, Fig.10 A schematic diagram of a reference three-dimensional face model provided in an embodiment of the present application, Fig.11 A schematic diagram of a second facial image provided in an embodiment of the present application, Fig.12 The schematic diagram of the target object three-dimensional face model provided in the embodiment of the present application, at this time, the expression formula of the reference three-dimensional face model is used to construct the prior information Fig.10 The reference 3D face model is then converted into Fig.10 The reference face 3D model and Fig.11 By fitting the second face image in , we can get Fig.12 The 3D face model of the target object in the image is understandable. Fig.12 When the three-dimensional face model shown has no expression, it can be used as a neutral expression base of the face. Fig.12 A side view of a three-dimensional face model is shown.

[0108] Step 280: construct multiple personalized facial expression bases according to predetermined expression base parameters and a neutral facial expression base, wherein each expression base parameter corresponds to a personalized facial expression base.

[0109] Exemplarily, since the expression base parameters corresponding to each benchmark personalized expression base are determined in advance based on the benchmark neutral identity information, in this step, the face personalized expression base of the face neutral expression base can be directly obtained based on the expression base parameters. In one embodiment, after the expression base parameters are added to the face neutral expression base, the corresponding face personalized expression base can be obtained. At this time, the calculation formula of the face personalized expression base is:

[0110] S_B i =M id +ΔBi

[0111] Among them, M id Represents the neutral facial expression base, S_B i represents the personalized expression base of the i-th face, ΔB i It represents the expression base parameters required to determine the personalized expression base of the i-th face, 1≤i≤n, and n is the total number of personalized expression bases (or expression base parameters) of faces.

[0112] Step 290: Acquire a second image collected online, where the second image includes a second facial image of the target object.

[0113] Step 2100: construct a three-dimensional face model of the target object based on a neutral face expression base and multiple personalized face expression bases.

[0114] In one embodiment, a weight coefficient is set for each face personalized expression base, and then each face personalized expression base is linearly weighted with the face neutral expression base in combination with the weight coefficient, so that a three-dimensional face model of the target object with expression can be obtained. At this time, the three-dimensional face model is represented as: Among them, B represents the 3D face model, M id Represents the neutral facial expression base, S_B i represents the personalized facial expression base of the i-th person, 1≤i≤n, n is the total number of personalized facial expression bases, β i S_B i The corresponding weight coefficient. It can be understood that S_B i -M id It can also be understood as the facial personalized expression base S_B i The corresponding expression base parameter ΔB i It should be noted that when constructing the three-dimensional face model currently, the weight coefficient used for each personalized facial expression base is the initialized weight coefficient, rather than the weight coefficient representing the expression of the target object in the second facial image. Therefore, the expression represented by the currently constructed three-dimensional face model is different from the expression of the second facial image.

[0115] Step 2110: Determine an error parameter formula when the three-dimensional face model is mapped to the second face image.

[0116] Exemplarily, the error parameter formula can also be understood as an energy functional, and its specific construction rule embodiment is not limited. In one embodiment, the error parameter formula is constructed by minimizing the residual. At this time, the error parameter formula can be expressed as:

[0117]

[0118] Among them, E represents the error parameter, B represents the 3D face model, and Bk represents the kth face key point in the three-dimensional face model (which can be understood as the three-dimensional coordinates of the kth face key point), 1≤k≤M, M is the total number of face key points (M=68 in the above embodiment), M id Represents the neutral facial expression base of the target object, S_B i represents the i-th personalized facial expression base of the target object, 1≤i≤n, n is the total number of personalized facial expression bases, β i S_B i The corresponding weight coefficient, f k represents the kth facial key point in the second facial image (which can be understood as the two-dimensional coordinates of the kth facial key point), s represents the scale factor when the three-dimensional face model is mapped to the second facial image, R represents the rigid rotation matrix when the three-dimensional face model is mapped to the second facial image, t represents the translation vector when the three-dimensional face model is mapped to the second facial image, s, R and t are pose parameters, and ||*|| represents the norm of *. It can be understood that through sR·B k +tf k It can reflect the difference between the kth facial key point in the 3D face model and the kth facial key point in the second face image when the 3D face model is mapped to the 2D plane, and then the error parameter is obtained according to the difference. In the above formula, β i , s, R and t are unknown. It should be noted that the facial key points of the 3D face model and the facial key points of the face image have the same arrangement order.

[0119] In another embodiment, the error parameter formula is constructed by the linear least squares method, that is, the above-mentioned minimized residual is converted into a form to be solved by the linear least squares method. Among them, the least squares method (also known as the least square method) is a mathematical optimization technique, which can be used to easily obtain unknown data (in the embodiment, β i , s, R and t are unknown data), and the sum of squares of the errors between these obtained data and the actual data is minimized. At this time, when the minimized residual is converted into the form of linear least squares solution, the error parameter formula can be expressed as:

[0120] minE exp =min||Aβ-b|| 2

[0121] Among them, E exp represents the error parameter, β represents the weight coefficient vector, β=(β1β2…β n ), n represents the total number of personalized facial expression bases, β irepresents the weight coefficient of the personalized expression base of the i-th face of the target object, 1≤i≤n, A=sR·ΔB, s represents the scale scaling factor when the three-dimensional face model is mapped to the second face image, R represents the rigid rotation matrix when the three-dimensional face model is mapped to the second face image, ΔB=(ΔB1ΔB2…ΔB n ), ΔB i represents the expression base parameter corresponding to the personalized expression base of the i-th face, b = ft-sR·M id , f represents the facial key points in the second face image (which can be understood as the two-dimensional coordinates of the facial key points), f = (f 1 … f M ), M is the total number of facial key points, t represents the translation vector when the 3D face model is mapped to the second face image, s, R and t are pose parameters, M id Represents the neutral facial expression base of the target object. From the above formula, we can see that Aβ can reflect the difference between the personalized facial expression base and the neutral facial expression base in the two-dimensional plane, and b can reflect the difference between the neutral facial expression base and the facial image in the two-dimensional plane. It can be understood that the closer the three-dimensional face model is to the face image, the smaller the difference between Aβ and b.

[0122] Step 2120: Determine the posture parameters of the three-dimensional face model and the weight coefficients of the personalized expression bases of each face when the error parameter is minimized according to the error parameter formula.

[0123] Exemplarily, after constructing the error parameter formula, since the unknowns in the error parameter formula include the posture parameter and the weight coefficient, the error parameter formula can be used to determine the posture parameter and the weight parameter used in the error parameter formula when the error parameter is minimum, and then the posture parameter and the weight parameter are used as the posture parameter and the weight coefficient obtained by the final calculation.

[0124] In one embodiment, the posture parameters and weight coefficients are calculated by alternating optimization. For example, the error parameter formula is: As an example, the process of alternately optimizing and calculating the pose parameters and weight coefficients is described. During the alternate optimization calculation, step 2120 may specifically include steps 2121 to 2126:

[0125] Step 2121: Obtain the initialization weight coefficient of each personalized facial expression base, and use the initialization weight coefficient as the current weight coefficient.

[0126] Exemplarily, the initialization weight coefficient refers to a pre-set weight coefficient, that is, a weight coefficient is pre-set for each face personalized expression base. The specific value of the initialization weight coefficient can be set according to actual conditions, such as selecting a value boundary as the initialization weight coefficient of the face personalized expression base according to the value range of the weight coefficient of the face personalized expression base.

[0127] In order to facilitate the description of the calculation process of the weight coefficient and posture parameters, the weight coefficient currently used is recorded as the current weight coefficient in the embodiment. It can be understood that since the initialization weight coefficient is used when the calculation starts, the initialization weight coefficient is first used as the current weight coefficient.

[0128] Step 2122: Substitute the current weight coefficient into the error parameter formula, and calculate the alternative pose parameters of the three-dimensional face model when the error parameter is minimized.

[0129] Substitute the current weight coefficient into the error parameter formula so that the weight coefficient in the error parameter formula is a fixed value (the value of the current weight coefficient). At this time, for the error parameter formula, the only unknown number is the posture parameter. In one embodiment, the weight coefficient of each face personalized expression base is set to 0. At this time, the error parameter formula Can be simplified to Among them, M id k Represents the kth facial key point in the neutral facial expression base (which can be understood as the three-dimensional coordinates of the facial key point). At this time, for the simplified error parameter formula, the only unknowns are s, R, and t. Afterwards, the simplified error parameter formula is calculated using the gold standard algorithm to solve the affine transformation matrix used when the error parameter between the three-dimensional face model and the second face image is minimized when the three-dimensional face model is mapped to the two-dimensional space, where the affine transformation refers to a linear transformation in geometry in a vector space (such as the space where the three-dimensional face model is located) and a translation transformation to another vector space (such as the space where the second face image is located). The affine transformation matrix refers to the matrix required for affine transformation. The gold standard algorithm can estimate the correspondence between two-dimensional coordinates (such as the coordinates of the facial key points in the second face image) and three-dimensional coordinates (such as the coordinates of the three-dimensional key points in the three-dimensional face model). The correspondence can also be understood as an affine transformation relationship. At this time, the estimated correspondence is the calculated affine transformation matrix. Afterwards, s, R, and t can be decomposed and derived through the affine transformation matrix, that is, the posture parameters are obtained. In the embodiment, the posture parameters calculated this time are recorded as candidate posture parameters. The candidate posture parameters can be understood as intermediate values, and the purpose of calculating the candidate posture parameters is to obtain the final posture parameters.

[0130] Step 2123: Substitute the candidate posture parameters into the error parameter formula, and calculate the candidate weight coefficients of each face personalized expression base when the error parameter is minimum.

[0131] Exemplarily, after calculating the candidate pose parameters, the currently calculated candidate pose parameters are substituted into the error parameter formula so that the pose parameters in the error reference formula are fixed values. At this time, for the error parameter formula, the only unknown number is the weight coefficient. After that, it can be simplified to Among them, in the simplified error parameter formula, the unknown number to be calculated is the weight coefficient β i . Afterwards, calculation is performed according to the error parameter formula to determine the specific value of the weight coefficient when the error parameter is minimum in the current calculation process. In the embodiment, the weight coefficient obtained by this calculation is recorded as the alternative weight coefficient. Among them, the alternative weight coefficient can be understood as an intermediate value, and the purpose of calculating the alternative weight coefficient is to obtain the final weight coefficient.

[0132] Step 2124: Update the current number of iterations.

[0133] Exemplarily, an iterative calculation process refers to the process of substituting the current weight coefficient into the error parameter formula to obtain the alternative posture parameters and the alternative weight coefficient. After obtaining the alternative posture parameters and the alternative weight coefficient, it is determined that an iterative calculation is completed, and the number of iterations is updated, that is, the current number of iterations is increased by 1. It can be understood that each time the alternative weight coefficient is obtained, the number of iterations is increased by 1, and the alternative weight coefficient and the alternative posture parameter obtained by the most recent iterative calculation are used as the current last obtained alternative weight coefficient and alternative posture parameter.

[0134] Step 2125: Determine whether the number of iterations reaches the number threshold. If the number of iterations does not reach the number threshold, use the candidate weight coefficient as the current weight coefficient and return to step 2122. If the number of iterations reaches the number threshold, execute step 2126.

[0135] In the embodiment, the number threshold is used to confirm whether to stop the iterative calculation. The number threshold can be set in combination with the actual situation, for example, the appropriate number threshold is determined in combination with historical experience data. Exemplarily, after updating the number of iterations, it is determined whether the current number of iterations reaches the number threshold. If so, the iterative calculation is stopped and step 2126 is executed. If not, the iterative calculation is continued. At this time, the alternative weight coefficient is used as the current weight coefficient, and the execution returns to step 2122 to start a new iterative calculation.

[0136] Step 2126: Use the finally obtained candidate pose parameters as the pose parameters of the three-dimensional face model, and use the finally obtained candidate weight coefficient as the weight coefficient of the personalized facial expression base.

[0137] The final candidate pose parameters and candidate weight coefficients refer to the candidate pose parameters and candidate weight coefficients obtained by the most recent iteration when the number of iterations reaches the number threshold. When the number of iterations reaches the number threshold, the iterative calculation is stopped, and the final candidate pose parameters and candidate weight coefficients are used as the pose parameters of the final 3D face model and the weight coefficients of the personalized facial expression base.

[0138] It should be noted that, generally speaking, when the number of iterations meets the threshold, after adjusting the three-dimensional face model according to the final alternative posture parameters and alternative weight coefficients, when the adjusted three-dimensional face model is mapped to a two-dimensional space, the obtained two-dimensional image is highly similar or identical to the second face image.

[0139] It can be understood that the above description is based on an example of fixing the weight coefficient first. In actual applications, the posture parameters can also be fixed first for calculation.

[0140] It should be noted that in practical applications, other calculation methods can also be used, and the embodiment does not limit this. It is only necessary to obtain the corresponding posture parameters and weight coefficients when the error parameters are minimized.

[0141] Step 2130: Send the posture parameters and weight coefficients to the remote device, so that the remote device generates a virtual image corresponding to the second facial image according to the posture parameters and weight coefficients.

[0142] In the above, by obtaining the pre-constructed reference neutral identity information, the reference neutral expression base and multiple reference personalized expression bases, then, according to the deformation information between the reference neutral expression base and the reference personalized expression base, each reference personalized expression base corresponding to the reference neutral identity information is determined, then, according to the reference personalized expression and the reference neutral identity information, the expression base parameters are determined, then, a first image including the first face image of the target object is obtained, the face neutral expression base of the target object is constructed by combining the reference neutral identity information, the first face image and the first coordinate array in the first face image, and the face personalized expression base corresponding to the face neutral expression base is determined by combining the expression base parameters, so that the difference between the neutral expression and the personalized expression is reflected by the expression base parameters. At this time, the expression base parameters only need to be constructed once. When the target object changes, there is no need to repeatedly construct the expression base parameters, and the expression base parameters can be repeatedly used to determine the face personalized expression base. In addition, when the face personalized expression base is determined by using the expression base parameters, there is no need to calculate the complex deformation transfer (that is, there is no need to calculate the deformation information between the face neutral expression base and the face personalized expression base), so that the face personalized expression base is constructed simply and quickly. In addition, the basic expressions are refined by FACS, and each basic expression corresponds to a reference personalized expression base, so that the expressions generated after the personalized expression base is superimposed are richer. In particular, the basic expressions defined by FACS are mainly divided into left-right symmetrical expressions. When the expression of the face image is an asymmetrical expression, it can be effectively captured and driven so that the obtained posture parameters and weight coefficients are close to the real face image. After that, the second image collected online is obtained, and the three-dimensional face model of the target object is constructed according to the neutral face expression base and the personalized face expression base, and the error parameter formula between the three-dimensional face model and the face image is constructed. After that, the weight coefficient of the personalized face expression base and the posture parameter of the three-dimensional face model when the error parameter is the smallest are determined according to the error parameter formula, and the weight coefficient and posture parameter are sent to the remote device so that the remote device generates the corresponding virtual image, which solves the technical problems of information leakage and jamming caused by transmitting real face images in related technologies, and ensures the imaging quality of the remote device. When calculating the weight coefficient and posture parameters, the error parameter formula can be converted into a linear solution formula by fixing the weight coefficient or posture parameter and performing iterative calculation, which simplifies the calculation process. In addition, the personalized facial expression base is refined by FACS, which can ensure that the obtained posture parameters and weight coefficients are close to the real second face image.

[0143] Fig.13 A flowchart of another virtual image construction method provided for an embodiment of the present application. This embodiment is specific based on the above embodiment. Fig.13 , the virtual image construction method specifically includes:

[0144] Step 310: Obtain pre-constructed baseline neutral identity information, a reference neutral expression base, and multiple reference personalized expression bases.

[0145] Step 320: Determine deformation information between the reference neutral expression base and each reference personalized expression base.

[0146] Step 330: Determine a plurality of reference personalized expression bases relative to the reference neutral identity information according to the deformation information and the reference neutral identity information, wherein each reference personalized expression base corresponds to a reference personalized expression base.

[0147] Step 340: Determine expression base parameters between the reference personalized expression base and the reference neutral identity information, where each reference personalized expression base corresponds to one expression base parameter.

[0148] Step 350: Acquire a first image collected offline, where the first image includes a first face image of the target object;

[0149] Step 360: construct a neutral facial expression base and multiple personalized facial expression bases of the target object according to the first facial image and predetermined expression base parameters.

[0150] Step 370: Obtain a pre-built mutually exclusive expression base.

[0151] Mutually exclusive expression bases refer to facial personalized expression bases whose corresponding expressions cannot appear on a human face at the same time. For example, chin left shift and chin right shift cannot appear on a human face at the same time. Therefore, the corresponding two facial personalized expression bases can be considered as mutually exclusive expression bases. For example, Fig.14 A schematic diagram of mutually exclusive expressions provided in the embodiment of the present application, referring to Fig.14 , based on the reader's vision, the lips and chin of the left face's personalized expression base are both moved to the left, and the lips and chin of the right face's personalized expression base are both moved to the right. For a human face, it can only make one of the expressions, but not two expressions at the same time. For another example, Fig.15 Another mutually exclusive expression schematic diagram provided in the embodiment of the present application is shown in FIG. Fig.15 , based on the reader's vision, the left corner of the mouth in the personalized expression of the face on the left is bent upward, and the left corner of the mouth in the personalized expression of the face on the right is bent downward. For a face, it can only make one of the expressions, but not two expressions at the same time. For another example, Fig.16 Another mutually exclusive expression schematic diagram provided in the embodiment of the present application, refer to Fig.16 , the expression of the left personalized facial expression base is opening the mouth, and the expression of the right personalized facial expression base is puffing up the cheeks. For a face, it is impossible to puff up the cheeks when opening the mouth, so it can be considered that they are mutually exclusive expression bases.

[0152] It can be understood that the expressions that cannot appear at the same time in the mutually exclusive expression bases not only refer to the basic expressions corresponding to each personalized expression base, but also include the superimposed expressions. At this time, when the superimposed expressions cannot appear on the human face at the same time, the corresponding multiple facial personalized expression bases are also mutually exclusive expression bases. For example, the superimposed expression is a frowning nose and a frowning left eyebrow, and another superimposed expression is a raised eyebrow and a raised left eyebrow tail. At this time, the two superimposed expressions cannot appear on the human face at the same time. Therefore, the facial personalized expression base of frowning nose and frowning left eyebrow and the facial personalized expression base of raising eyebrows and raising left eyebrow tail are mutually exclusive expression bases.

[0153] In one embodiment, taking the customized basic expressions in Table 1 as an example, mutually exclusive expression bases among the 26 personalized expression bases corresponding to the basic expressions are determined. It is understandable that mutually exclusive expression bases can be constructed manually, and the virtual image construction device directly obtains the mutually exclusive expression bases. The virtual image construction device can also gradually add each basic expression to the same three-dimensional face model and superimpose each basic expression to determine whether the three-dimensional face model can display each expression at the same time, and then determine the mutually exclusive expression base.

[0154] In one embodiment, the mutually exclusive expression bases among the 26 personalized facial expression bases are shown in the following table:

[0155] Mutex 1 Mutex 2 B0 B2 B1 B3 B4,B25 B6,B7 B5,B25 B6,B8 B9 B24 B10 B11 B12 B21 B13 B22

[0156] Table 2

[0157] In the above table, the personalized expression bases included in mutually exclusive 1 and the personalized expression bases included in mutually exclusive 2 in the same row are mutually exclusive expression bases. It should be noted that "B" in Table 2 corresponds to "Blendshape" in Table 1, and the number after "B" in Table 2 is the number of "Blendshape".

[0158] Step 380: grouping the personalized expression bases of each human face according to the mutually exclusive expression bases to obtain a plurality of expression base groups, wherein any two personalized expression bases of human faces in each expression base group are not mutually exclusive.

[0159] In one embodiment, the face personalized expression bases are grouped according to mutually exclusive expression bases, and each group is recorded as an expression base group. At this time, the face personalized expression bases in each expression base group are not mutually exclusive. For example, if an expression base group contains the face personalized expression base corresponding to B1, then it will not contain the face personalized expression base corresponding to B3. If an expression base group contains the face personalized expression bases corresponding to B4 and B25, then it will not contain the face personalized expression bases corresponding to B6 and B7.

[0160] It is understandable that the specific manner of grouping is not limited in the embodiment, as long as each expression base group does not contain mutually exclusive personalized facial expression bases.

[0161] It should be noted that the more mutually exclusive expression bases there are, the more expression base groups there are. For example, in Table 2, there are 8 mutually exclusive expression bases. At this time, when the 26 facial personalized expression bases are grouped according to the 8 mutually exclusive expression bases, 2 8 In this case, the number of expression basis sets is too large, which is not conducive to subsequent calculations. For example, in the subsequent calculation process, it only takes 1ms to obtain the corresponding weight coefficient and posture parameter based on one expression basis set, but it takes 1ms to obtain the corresponding weight coefficient and posture parameter based on 2 expression basis sets. 8 It takes a long time to calculate each expression base group, which is not conducive to the real-time requirements of video communication. Therefore, in the embodiment, after obtaining each expression base group, each expression base group is screened to reduce the number of expression base groups, thereby reducing the subsequent calculation time. In one embodiment, the following schemes may be included when screening each expression base group:

[0162] Solution 1, after executing step 380, also includes: determining a second coordinate array of each facial key point in the second facial image; determining a first candidate facial personalized expression base of the second facial image based on the coordinates of the facial key points associated in the second coordinate array; determining a first exclusionary facial personalized expression base corresponding to the first candidate facial personalized expression base based on mutually exclusive expression bases; and deleting the expression base group including the first exclusionary facial personalized expression base from multiple expression base groups.

[0163] It can be understood that when a certain expression appears in the second face image, the coordinate distance of each corresponding face key point or the relative position relationship of the face key points should satisfy certain constraints. For example, Figure 8 , for the mouth position, when the mouth of the target object is open, there should be a certain distance between the face key point P1 and the face key point P4, there should be a certain distance between the face key point P2 and the face key point P5, and there should be a certain distance between the face key point P3 and the face key point P6. It can be understood that when the mouth of the target object is closed, the face key point P1 and the face key point P4 should overlap or have a small distance, the face key point P2 and the face key point P5 should overlap or have a small distance, and the face key point P3 and the face key point P6 should overlap or have a small distance. Therefore, by the distance between the above-mentioned face key points, it can be determined whether the mouth is closed or open, and then the face personalized expression base that will not appear in the second face image can be determined based on the state of the mouth, so as to filter the expression base group in combination with the face personalized expression base that will not appear in the second face image. Therefore, the face key points of the target object in the second face image are used as prior information to reduce the number of expression base groups.

[0164] Exemplarily, the coordinates of each key point of the face in the second face image are first determined, and a key point coordinate array is formed. In the embodiment, the key point coordinate array obtained based on the second face image is recorded as the second coordinate array. It can be understood that the determination method of the second coordinate array is the same as the determination method of the first coordinate array, which will not be repeated here. It should be noted that in the process of video communication, there is a certain correlation between the second images of two adjacent frames, and the second coordinate array is obtained based on one frame of the second image. At this time, if the coordinates of the key points of the face with the same face semantic information in the two adjacent frames of the second image have a large difference, it will affect the later calculation, so that the final generated virtual image has a jitter problem. In order to prevent the above problems, in the embodiment, after obtaining the second coordinate array, the second coordinate array can also be filtered and smoothed. Among them, the filtering operation refers to adjusting the second coordinate array of the current second image in combination with the second coordinate array in the previous frame of the second image, ensuring that the second coordinate array of the previous frame of the second image to the second coordinate array of the current second image is smooth and gradual, so that the second coordinate arrays of each frame in the video communication process are smooth and gradual. In one embodiment, the filtering operation is realized by Kalman filtering. When the Kalman filter is performed on the second coordinate data of the current frame, the second coordinate array of the current frame and the second coordinate array of the previous frame are weighted to update the weighted result to the second coordinate data of the current frame. The smoothing operation is used to avoid the situation where some facial key points are outliers, so that the coordinate curve between adjacent facial key points is smooth. In one embodiment, the PCA algorithm is used to smooth the filtered second coordinate array to update the second coordinate array. It can be understood that the second coordinate array used subsequently is the second coordinate array after the filtering operation and the smoothing operation.

[0165] Afterwards, the possible personalized expressions in the second face image are determined according to the second coordinate array. At this time, the facial personalized expression base corresponding to the possible personalized expression is recorded as the first candidate facial personalized expression base. In one embodiment, the facial key points required for determining the personalized expression can be pre-set, and the required facial key points can be used as the associated facial key points, and then the possible facial personalized expression base can be inferred according to the associated facial key points. For example, referring to Figure 8 , through the position difference between the face key points P1 and P4, the position difference between the face key points P2 and P5, and the position difference between the face key points P3 and P6 of the target object's mouth, the position difference between the upper lip and the lower lip in the mouth can be determined, and then the mouth can be determined to be closed or open. The position difference of the mouth can be expressed as p1, p2, p3, p4, p5 and p6 represent the two-dimensional coordinates of the corresponding facial key points, and then the state of the mouth is determined by setting the position difference threshold. For example, the position difference threshold corresponding to the mouth is 3. At this time, when d>3, it is determined that the mouth of the target object in the second facial image is open. At this time, the first candidate facial personalized expression base that may appear in the second facial image is the facial personalized expression base corresponding to the open mouth (corresponding to B9 in Table 1). For another example, for the eye area, the left eye and the right eye each correspond to 6 facial key points, among which there are two facial key points in the upper eye socket and the lower eye socket. Similar to the mouth closure, when the eyes are closed, the corresponding facial key points in the upper eye socket and the lower eye socket will be close to or overlap. Therefore, by the position difference of the facial key points in the upper eye socket and the lower eye socket, it can be determined whether the eyes are open or closed. Then the first candidate facial personalized expression base that may appear in the second facial image is determined.

[0166] After determining the first candidate face personalized expression base, the mutually exclusive face personalized expression base of the first candidate face personalized expression base can be determined according to the preset mutually exclusive expression bases. In the embodiment, the mutually exclusive face personalized expression base of the first candidate face personalized expression base is recorded as the first exclusive face personalized expression base. For example, when the first candidate face personalized expression base is an open mouth, the first exclusive face personalized expression base is a face personalized expression base corresponding to a bulging cheek. Afterwards, in each expression base group, the expression base group including the first exclusive face personalized expression base is deleted to delete the face personalized expression base that is impossible to appear in the second face image, thereby realizing the screening of the expression base group.

[0167] Solution 2, after executing step 380, also includes: determining a second coordinate array of each facial key point in the second facial image and a third coordinate array of each facial key point in the neutral facial expression base; determining a pose estimation parameter of the target object in the second facial image based on the third coordinate array and the second coordinate array; transforming the neutral facial expression base based on the pose estimation parameters to obtain an estimated facial expression base; mapping the estimated facial expression base to a two-dimensional space to obtain a fourth coordinate array of each facial key point in the estimated facial expression base in the two-dimensional space; determining a second candidate facial personalized expression base for the second facial image based on the fourth coordinate array and the second coordinate array; determining a second exclusionary facial personalized expression base corresponding to the second candidate facial personalized expression base based on mutually exclusive expression bases; and deleting the expression base group containing the second exclusionary facial personalized expression base from multiple expression base groups.

[0168] It can be understood that when a certain expression appears in the second facial image, there will be a difference in the position of the facial key points in its key parts and the facial key points in the face when the target object has a neutral expression. For example, for some facial key points of the left eyebrow, when the target object raises the left eyebrow, the facial key point should be above the facial key point in the neutral expression, and when the target object frowns the left eyebrow, the facial key point should be below the facial key point in the neutral expression. Therefore, the relative position relationship between the facial key points of the second facial image and the same facial key points of the target object with a neutral expression (i.e., the deviation between the facial key points) can be used to determine the facial personalized expression bases that may appear in the second facial image, and then determine the facial personalized expression bases that will not appear in the second facial image, so as to screen the expression base groups in combination with the facial personalized expression bases that will not appear in the second facial image, thereby reducing the number of expression base groups.

[0169] Exemplarily, the coordinates of each facial key point in the second facial image are first determined to obtain a second coordinate array. It can be understood that the second coordinate array is determined in the same manner as the second coordinate array in Scheme 1, and will not be described in detail here. In addition, the coordinates of each facial key point in the facial neutral expression base are determined, and the coordinates are three-dimensional coordinates. After that, the coordinates of each facial key point are combined into a key point coordinate array. In the embodiment, the key point coordinate array composed of each facial key point in the facial neutral expression base is recorded as a third coordinate array, wherein the order of each facial key point in the third coordinate array is the same as the order of each facial key point in the second coordinate array.

[0170] Afterwards, the pose information of the target object in the second face image is estimated through the third coordinate array and the second coordinate array. Optionally, each three-dimensional coordinate in the third coordinate array is mapped to a two-dimensional space to obtain the two-dimensional coordinate corresponding to each three-dimensional coordinate. Afterwards, the difference between the two-dimensional coordinate and the coordinate of the same facial key point in the second coordinate array is determined, and the rigid rotation matrix, scale scaling factor and translation vector used when the difference is the smallest after the third coordinate array is mapped are determined, and then the pose parameters of the target object in the second face image are estimated. In an embodiment, the currently estimated pose parameters are recorded as pose estimation parameters. Afterwards, the neutral facial expression basis is mapped to a two-dimensional space according to the pose estimation parameters to obtain the two-dimensional coordinates of each facial key point in the third coordinate array in the two-dimensional space, and form a new key point coordinate array. In an embodiment, the key point coordinate array of the third coordinate array in the two-dimensional space is recorded as the fourth coordinate array.

[0171] Afterwards, the personalized expressions that may appear in the second facial image are determined based on the second coordinate array and the fourth coordinate array. At this time, the facial personalized expression base corresponding to the personalized expression that may appear is recorded as the second candidate facial personalized expression base. In one embodiment, the facial key points required for determining the personalized expression can be pre-set, and then the facial personalized expression base that appears can be inferred based on the facial key points. Exemplarily, when pre-determining each personalized expression, the relative position relationship between each facial key point in the second facial image and the same facial key point in the facial neutral expression base is determined, and then the second coordinate array and the fourth coordinate array are used to determine whether the facial key point satisfies the relative position relationship, and then the second candidate facial personalized expression base that may appear in the second facial image is determined. For example, Fig.17 The fourth schematic diagram of key points of a face provided in an embodiment of the present application. Fig.17 , based on the reader's perspective, it identifies the facial key points in the second face image and the positions of the facial key points in the second face image after mapping the facial neutral expression base according to the pose estimation parameters. At this time, for the left eyebrow, face key point P1 and face key point P4 are face key points representing the same semantics, face key point P2 and face key point P5 are face key points representing the same semantics, face key point P3 and face key point P6 are face key points representing the same semantics, face key point P1, face key point P2 and face key point P3 are positions in the second face image, face key point P4, face key point P5 and face key point P6 are positions of each face key point in the second face image after mapping the face neutral expression base according to the pose estimation parameters, and the relative position relationship between face key point P1 and face key point P4, the relative position relationship between face key point P2 and face key point P5, and the relative position relationship between face key point P3 and face key point P6 can be used to determine whether the target object raises the left eyebrow or frowns the left eyebrow. It can be understood that when the target object raises the left eyebrow, the facial key point P1 should be located above the facial key point P4 under the neutral expression, the facial key point P2 should be located above the facial key point P5 under the neutral expression, and the facial key point P3 should be located above the facial key point P6 under the neutral expression. At this time, the formula can be used: to determine the relative position of the key points of the face, p1, p2, p3, p4, p5 and p6 respectively represent the two-dimensional coordinates of the corresponding key points of the face, wherein, when d>0, it is determined that the target object in the second face image raises the left eyebrow. At this time, the second candidate facial personalized expression base that may appear in the second face image is the facial personalized expression base related to raising the left eyebrow (corresponding to B6 and B7 in Table 1).

[0172] After determining the second candidate face personalized expression base, the mutually exclusive face personalized expression base of the second candidate face personalized expression base can be determined according to the preset mutually exclusive expression base. In the embodiment, the mutually exclusive face personalized expression base of the second candidate face personalized expression base is recorded as the second exclusive face personalized expression base. For example, when the second candidate face personalized expression base includes raising eyebrows and raising the tail of the left eyebrow, the second exclusive face personalized expression base is the face personalized expression base corresponding to frowning the left eyebrow and wrinkling the nose. Afterwards, in each expression base group, the expression base group including the second exclusive face personalized expression base is deleted to delete the face personalized expression base that cannot appear in the second face image, thereby realizing the screening of the expression base group.

[0173] It is understandable that step 380 may also be performed after step 390, step 3100, and step 3110, and the embodiment does not limit this.

[0174] Step 390: Acquire a second image collected online, where the second image includes a second facial image of the target object.

[0175] Step 3100: construct a three-dimensional face model of the target object based on a neutral face expression base and multiple personalized face expression bases.

[0176] It is understandable that when constructing a three-dimensional face model, all the personalized facial expression bases are required.

[0177] Step 3110: Determine an error parameter formula when the three-dimensional face model is mapped to the second face image.

[0178] The execution process of step 3110 is the same as that of step 2110.

[0179] Step 3120: Calculate the minimum error parameter corresponding to each expression basis group according to the error parameter formula, as well as the posture parameters of the three-dimensional face model and the weight coefficient of each personalized expression basis in the expression basis group when the error parameter is the minimum.

[0180] In one embodiment, for each currently retained expression base group, the calculation is performed in units of the expression base group, and one expression base group is optimized each time. Since the calculation process of each expression base group is the same, the embodiment is described by taking the calculation of one expression base group as an example. Exemplarily, according to the error parameter formula, when calculating the minimum error parameter, the weight coefficients of each face personalized expression base in the expression base group and the posture parameters of the three-dimensional face model. Since the expression base group does not contain all the face personalized expression bases, during the calculation process, the weight coefficients of the face personalized expression bases not included in the expression base group can be always set to 0 to reduce the number of weight coefficients to be solved. It can be understood that in the calculation process, the iterative calculation method can also be used. For details, refer to the process described in step 2120. The only difference is that the weight coefficients of the face personalized expression bases not included in the expression base group in the final weight coefficients are 0.

[0181] It is understandable that, since each expression basis group contains different personalized facial expression bases, when using different expression basis groups for calculation, the minimum error parameter, weight coefficient and posture parameter obtained may be different. Therefore, after calculating each expression basis group in the above manner, each expression basis group corresponds to a minimum error parameter, weight coefficient and posture parameter.

[0182] Step 3130: Among the minimum error parameters corresponding to each expression basis set, the posture parameters and weight coefficients corresponding to the smallest minimum error parameters are used as the final posture parameters and weight coefficients.

[0183] The smaller the minimum error parameter is, the closer the two-dimensional image obtained when mapping the three-dimensional face model to the two-dimensional space is to the second face image. Therefore, in the embodiment, among the minimum error parameters corresponding to each expression basis set, the smallest minimum error parameter is selected. Generally speaking, there is only one smallest minimum error parameter. Afterwards, the expression basis set corresponding to the smallest minimum error parameter is confirmed, and the posture parameters and weight coefficients of the expression basis set are obtained as the final posture parameters and weight coefficients. It can be understood that after adjusting the three-dimensional face model using the posture parameters and weight coefficients corresponding to the smallest minimum error parameter, the three-dimensional face model is closest to the second face image.

[0184] Step 3140: Send the posture parameters and weight coefficients to the remote device, so that the remote device generates a virtual image corresponding to the second facial image according to the posture parameters and weight coefficients.

[0185] In the above, by obtaining the pre-constructed reference neutral identity information, the reference neutral expression base and multiple reference personalized expression bases, then, according to the deformation information between the reference neutral expression base and the reference personalized expression base, the reference personalized expression base corresponding to the reference neutral identity information is determined, then, according to the reference personalized expression and the reference neutral identity information, the expression base parameters are determined, then, the first image including the first face image of the target object is obtained, the face neutral expression base of the target object is constructed by combining the reference neutral identity information, the first face image and the first coordinate array in the first face image, and the face personalized expression base corresponding to the face neutral expression base is determined by combining the expression base parameters, so that the difference between the neutral expression and the personalized expression is reflected by the expression base parameters. At this time, the expression base parameters only need to be constructed once. When the target object changes, there is no need to repeatedly construct the expression base parameters, and the expression base parameters can be reused. In addition, when the face personalized expression base is determined by using the expression base parameters, there is no need to calculate the complex deformation transfer (that is, there is no need to calculate the deformation information between the face neutral expression base and the face personalized expression base), so that the face personalized expression base is constructed simply and quickly. Afterwards, the second image collected online is obtained, and a three-dimensional face model of the target object is constructed based on the neutral face expression base and the personalized face expression base, and the error parameter formula between the three-dimensional face model and the face image is constructed. By finding mutually exclusive expression bases and grouping them according to the mutually exclusive expression bases, the pose parameters and weight coefficients are calculated in units of expression base groups, which reduces the number of weight coefficients solved each time, making the expression coefficient solution more accurate and efficient. At the same time, the expression base groups are screened in combination with the key points of the face, which can reduce the number of expression base groups, effectively and quickly narrow the search range of the expression base groups, and reduce the solution speed of the weight coefficient and pose parameter to about 13ms, meeting the real-time needs of video communication. Afterwards, the weight coefficient of the personalized face expression base when the error parameter is the smallest and the pose parameter of the three-dimensional face model are determined according to the error parameter formula, and the weight coefficient and pose parameter are sent to the remote device so that the remote device generates the corresponding virtual image, which can solve the technical problems of information leakage and jamming caused by transmitting real face images in related technologies, and ensure the imaging quality of the remote device.

[0186] Fig.18 A schematic diagram of a virtual image construction device provided in an embodiment of the present application, referring to Fig.18 The license plate character recognition device includes a first image acquisition module 401, an expression base construction module 402, a second image acquisition module 403, a parameter determination module 404 and a parameter sending module 405.

[0187] Among them, the first image acquisition module 401 is used to acquire a first image collected offline, and the first image includes a first facial image of a target object; the expression base construction module 402 is used to construct a neutral facial expression base and multiple personalized facial expression bases of the target object based on the first facial image and predetermined expression base parameters; the second image acquisition module 403 is used to acquire a second image collected online, and the second image includes a second facial image of the target object; the parameter determination module 404 is used to determine the posture parameters required when the three-dimensional facial model of the target object is mapped to the second facial image and the weight coefficients of each of the personalized facial expression bases based on the neutral facial expression base and the multiple personalized facial expression bases; the parameter sending module 405 is used to send the posture parameters and the weight coefficients to the remote device, so that the remote device generates a virtual image corresponding to the second facial image according to the posture parameters and the weight coefficients.

[0188] On the basis of the above embodiment, it also includes: an identity information acquisition module, which is used to obtain pre-constructed baseline neutral identity information, a reference neutral expression base and multiple reference personalized expression bases before obtaining the first image collected offline; a deformation information determination module, which is used to determine the deformation information between the reference neutral expression base and each reference personalized expression base; a baseline personalized determination module, which is used to determine multiple baseline personalized expression bases relative to the baseline neutral identity information according to the deformation information and the baseline neutral identity information, and each reference personalized expression base corresponds to a baseline personalized expression base; an expression base parameter determination module, which is used to determine the expression base parameter between the baseline personalized expression base and the baseline neutral identity information, and each baseline personalized expression base corresponds to an expression base parameter.

[0189] On the basis of the above-mentioned embodiment, the expression base construction module 402 includes: a first array determination unit, used to determine the first coordinate array of each facial key point in the first facial image; a neutral expression base construction unit, used to construct a neutral facial expression base of the target object according to the first coordinate array, the first facial image and the benchmark neutral identity information; a personalized expression base construction unit, used to construct multiple personalized facial expression bases according to predetermined expression base parameters and the neutral facial expression base, and each expression base parameter corresponds to a personalized facial expression base.

[0190] Based on the above embodiment, the parameter determination module 404 includes: a three-dimensional model construction unit, which is used to construct a three-dimensional face model of the target object based on a neutral face expression base and multiple personalized face expression bases; an error parameter determination unit, which is used to determine the error parameter formula when the three-dimensional face model is mapped to a second face image; and a parameter calculation unit, which is used to determine the posture parameters of the three-dimensional face model and the weight coefficients of each personalized face expression base when the error parameter is minimized according to the error parameter formula.

[0191] On the basis of the above embodiment, it also includes: a mutually exclusive acquisition module for acquiring pre-built mutually exclusive expression bases; a grouping module for grouping each face personalized expression base according to the mutually exclusive expression bases to obtain multiple expression base groups, and any two face personalized expression bases in each expression base group are not mutually exclusive. Correspondingly, the parameter calculation unit includes: a grouping calculation subunit for calculating the minimum error parameter corresponding to each expression base group according to the error parameter formula, as well as the posture parameters of the three-dimensional face model and the weight coefficients of each face personalized expression base in the expression base group at the minimum error parameter; a parameter selection subunit for taking the posture parameters and weight coefficients corresponding to the minimum minimum error parameters in the minimum error parameters corresponding to each expression base group as the final posture parameters and weight coefficients.

[0192] On the basis of the above embodiment, it also includes: a first determination module, which is used to group each face personalized expression base according to mutually exclusive expression bases, and after obtaining multiple expression base groups, determine the second coordinate array of each face key point in the second face image; a first alternative determination module, which is used to determine the first alternative face personalized expression base of the second face image according to the coordinates of the associated face key points in the second coordinate array; a first exclusion determination module, which is used to determine the first exclusion face personalized expression base corresponding to the first alternative face personalized expression base according to the mutually exclusive expression bases; and a first deletion module, which is used to delete the expression base group containing the first exclusion face personalized expression base from multiple expression base groups.

[0193] On the basis of the above embodiment, it also includes: a second determination module, which is used to group each face personalized expression base according to mutually exclusive expression bases, and after obtaining multiple expression base groups, determine the second coordinate array of each face key point in the second face image and the third coordinate array of each face key point in the face neutral expression base; a posture estimation module, which is used to determine the posture estimation parameters of the target object in the second face image according to the third coordinate array and the second coordinate array; a transformation module, which is used to transform the face neutral expression base according to the posture estimation parameters to obtain the face estimated expression base; a mapping module, which is used to map the face estimated expression base to a two-dimensional space to obtain a fourth coordinate array of each face key point in the face estimated expression base in the two-dimensional space; a second alternative determination module, which is used to determine a second alternative face personalized expression base of the second face image according to the fourth coordinate array and the second coordinate array; a second exclusion determination module, which is used to determine the second exclusion face personalized expression base corresponding to the second alternative face personalized expression base according to the mutually exclusive expression bases; and a second deletion module, which is used to delete the expression base group containing the second exclusion face personalized expression base from the multiple expression base groups.

[0194] On the basis of the above embodiment, the parameter calculation unit includes: an initialization acquisition subunit, which is used to obtain the initialization weight coefficient of each personalized expression base of the face, and use the initialization weight coefficient as the current weight coefficient; a first substitution subunit, which is used to substitute the current weight coefficient into the error parameter formula, and calculate the alternative posture parameters of the three-dimensional face model when the error parameter is minimum; a second substitution subunit, which is used to substitute the alternative posture parameters into the error parameter formula, and calculate the alternative weight coefficient of each personalized expression base of the face when the error parameter is minimum; an update subunit, which is used to update the current number of iterations; a number judgment subunit, which is used to judge whether the number of iterations reaches the number threshold; a return subunit, which is used to update the alternative weight coefficient to the current weight coefficient when the number of iterations does not reach the number threshold, and return to execute the operation of substituting the current weight coefficient into the error parameter formula until the number of iterations reaches the number threshold; a parameter determination subunit, which is used to use the finally obtained alternative posture parameters as the posture parameters of the three-dimensional face model when the number of iterations reaches the number threshold, and use the finally obtained alternative weight coefficient as the weight coefficient of the personalized expression base of the face.

[0195] Based on the above embodiment, the error parameter formula is: Among them, E represents the error parameter, B represents the 3D face model, and B k represents the kth facial key point in the 3D face model, 1≤k≤M, M is the total number of facial key points, M id Represents the neutral facial expression base of the target object, S_B i represents the i-th personalized facial expression base of the target object, 1≤i≤n, n is the total number of personalized facial expression bases, β i S_B i The corresponding weight coefficient, f k represents the kth facial key point in the second facial image, s represents the scale factor when the three-dimensional face model is mapped to the second facial image, R represents the rigid rotation matrix when the three-dimensional face model is mapped to the second facial image, t represents the translation vector when the three-dimensional face model is mapped to the second facial image, s, R and t are pose parameters; or, the error parameter formula is: minE exp =min||Aβ-b|| 2 , where E exp represents the error parameter, β represents the weight coefficient vector, β=(β1 β2 … β n ), n represents the total number of personalized facial expression bases, β iThe weight coefficient of the personalized expression base of the i-th face of the target object is 1≤i≤n, A=sR·ΔB, s represents the scale scaling factor when the three-dimensional face model is mapped to the second face image, R represents the rigid rotation matrix when the three-dimensional face model is mapped to the second face image, ΔB=(ΔB1 ΔB2… ΔB n ), ΔB i represents the expression base parameter corresponding to the personalized expression base of the i-th face, b = ft-sR·M id , f represents the facial key points in the second face image, t represents the translation vector when the 3D face model is mapped to the second face image, s, R and t are pose parameters, M id Represents the neutral facial expression base of the target object.

[0196] The virtual image construction device provided above can be used to execute the virtual image construction method provided by any of the above embodiments, and has corresponding functions and beneficial effects.

[0197] It is worth noting that in the embodiment of the above-mentioned virtual image construction device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0198] Fig.19 This is a schematic diagram of the structure of a virtual image construction device provided in an embodiment of the present application. Fig.19 As shown, the virtual image construction device includes a processor 50, a memory 51, an input device 52, and an output device 53; the number of the processor 50 in the virtual image construction device can be one or more. Fig.19 A processor 50 is taken as an example. The processor 50, the memory 51, the input device 52, and the output device 53 in the virtual image construction device can be connected via a bus or other means. Fig.19 The example of connecting through bus is taken in the following.

[0199] The memory 51, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the virtual image construction method in the embodiment of the present application (for example, the first image acquisition module 401, the expression base construction module 402, the second image acquisition module 403, the parameter determination module 404 and the parameter sending module 405 in the virtual image construction device). The processor 50 executes various functional applications and data processing of the virtual image construction device by running the software programs, instructions and modules stored in the memory 51, that is, realizes the above-mentioned virtual image construction method.

[0200] The memory 51 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the virtual image construction device, etc. In addition, the memory 51 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 51 may further include a memory remotely arranged relative to the processor 50, and these remote memories may be connected to the virtual image construction device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0201] The input device 52 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the virtual image construction device, and can also include an image acquisition device, an audio acquisition device, etc. The output device 53 can include a display device such as a display screen, and can also include an audio playback device. In addition, the virtual image construction device can also include a communication device to achieve data communication with a remote device, such as video communication.

[0202] The above-mentioned virtual image construction device includes a virtual image construction device, which can be used to execute any virtual image construction method and has corresponding functions and beneficial effects.

[0203] In addition, an embodiment of the present application also provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to perform relevant operations in the virtual image construction method provided in any embodiment of the present application, and have corresponding functions and beneficial effects.

[0204] Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0205] Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0206] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0207] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0208] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0209] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A virtual image construction method, characterized in that: include: Acquire a first image acquired offline, where the first image includes a first face image of a target object; Constructing a neutral facial expression base and a plurality of personalized facial expression bases of the target object according to the first facial image and predetermined expression base parameters; Acquire a second image collected online, where the second image includes a second face image of the target object; Determining, based on the neutral facial expression base and the plurality of personalized facial expression bases, the pose parameters required when the three-dimensional facial model of the target object is mapped to the second facial image and the weight coefficients of the personalized facial expression bases; Sending the posture parameters and the weight coefficients to a remote device, so that the remote device generates a virtual image corresponding to the second facial image according to the posture parameters and the weight coefficients; Wherein, determining the posture parameters required when the three-dimensional face model of the target object is mapped to the second face image and the weight coefficients of each of the personalized face expression bases according to the neutral face expression base and the plurality of personalized face expression bases includes: Constructing a three-dimensional face model of the target object according to the neutral face expression base and the plurality of personalized face expression bases; Determine an error parameter formula when the three-dimensional face model is mapped to the second face image; According to the error parameter formula, determining the posture parameters of the three-dimensional face model and the weight coefficients of each of the personalized facial expression bases when the error parameter is minimum; The step of constructing the three-dimensional face model of the target object according to the neutral face expression base and the plurality of personalized face expression bases includes: Each of the personalized facial expression bases and the neutral facial expression base are linearly weighted using an initialization weight coefficient to obtain a three-dimensional facial model of the target object.

2. The virtual image construction method according to claim 1, characterized in that: Before acquiring the first image acquired offline, the method includes: Obtaining pre-built baseline neutral identity information, a reference neutral expression base, and multiple reference personalized expression bases; Determining deformation information between the reference neutral expression base and each of the reference personalized expression bases; Determine, according to the deformation information and the reference neutral identity information, a plurality of reference personalized expression bases relative to the reference neutral identity information, each of the reference personalized expression bases corresponding to one of the reference personalized expression bases; An expression base parameter between the reference personalized expression base and the reference neutral identity information is determined, each reference personalized expression base corresponding to one expression base parameter.

3. The virtual image construction method according to claim 2, characterized in that: The step of constructing a neutral facial expression base and a plurality of personalized facial expression bases of the target object according to the first facial image and predetermined expression base parameters comprises: Determine a first coordinate array of each facial key point in the first facial image; Constructing a neutral facial expression base of the target object according to the first coordinate array, the first facial image and the reference neutral identity information; A plurality of personalized facial expression bases are constructed according to predetermined expression base parameters and the neutral facial expression base, and each of the expression base parameters corresponds to a personalized facial expression base.

4. The virtual image construction method according to claim 1, characterized in that: Also includes: Get a pre-built mutually exclusive expression base; Grouping the personalized facial expression bases according to the mutually exclusive expression bases to obtain a plurality of expression base groups, wherein any two personalized facial expression bases in each expression base group are not mutually exclusive; The step of determining the posture parameters of the three-dimensional face model and the weight coefficients of each of the personalized facial expression bases when the error parameter is minimum according to the error parameter formula includes: According to the error parameter formula, the minimum error parameter corresponding to each expression basis group, as well as the posture parameters of the three-dimensional face model and the weight coefficients of the personalized expression bases of each face in the expression basis group at the minimum error parameter are calculated; Among the minimum error parameters corresponding to each of the expression basis sets, the posture parameters and weight coefficients corresponding to the smallest minimum error parameters are taken as the final posture parameters and weight coefficients.

5. The virtual image construction method according to claim 4, characterized in that: After grouping the personalized facial expression bases according to the mutually exclusive expression bases to obtain a plurality of expression base groups, the method includes: Determine a second coordinate array of each facial key point in the second facial image; Determine a first candidate personalized facial expression base of the second facial image according to the associated facial key point coordinates in the second coordinate array; Determine, according to the mutually exclusive expression bases, a first exclusive face personalized expression base corresponding to the first candidate face personalized expression base; The expression base group including the first exclusionary face personalized expression base is deleted from the multiple expression base groups.

6. The virtual image construction method according to claim 4, characterized in that: After grouping the personalized facial expression bases according to the mutually exclusive expression bases to obtain a plurality of expression base groups, the method includes: Determine a second coordinate array of each facial key point in the second facial image and a third coordinate array of each facial key point in the facial neutral expression base; Determine a pose estimation parameter of the target object in the second face image according to the third coordinate array and the second coordinate array; Transforming the neutral facial expression base according to the pose estimation parameters to obtain an estimated facial expression base; Mapping the estimated face expression base into a two-dimensional space to obtain a fourth coordinate array of each face key point in the estimated face expression base in the two-dimensional space; Determine a second candidate face personalized expression base of the second face image according to the fourth coordinate array and the second coordinate array; Determine, according to the mutually exclusive expression bases, a second exclusive face personalized expression base corresponding to the second candidate face personalized expression base; The expression base group including the second exclusionary face personalized expression base is deleted from the plurality of expression base groups.

7. The virtual image construction method according to claim 1, characterized in that: The step of determining the posture parameters of the three-dimensional face model and the weight coefficients of each of the personalized facial expression bases when the error parameter is minimum according to the error parameter formula includes: Obtaining the initialization weight coefficient of each of the personalized facial expression bases, and using the initialization weight coefficient as the current weight coefficient; Substituting the current weight coefficient into the error parameter formula, and calculating the candidate pose parameters of the three-dimensional face model when the error parameter is minimum; Substituting the candidate posture parameters into the error parameter formula, and calculating the candidate weight coefficients of each of the personalized facial expression bases when the error parameter is minimum; Update the current number of iterations; Determine whether the number of iterations reaches a threshold number; When the number of iterations does not reach the number threshold, the candidate weight coefficient is updated to the current weight coefficient, and the operation of substituting the current weight coefficient into the error parameter formula is returned to be executed until the number of iterations reaches the number threshold; When the number of iterations reaches the number threshold, the finally obtained candidate posture parameters are used as the posture parameters of the three-dimensional face model, and the finally obtained candidate weight coefficient is used as the weight coefficient of the personalized facial expression base.

8. The virtual image construction method according to claim 1, characterized in that: The error parameter formula is: Among them, E represents the error parameter, B represents the 3D face model, and B k represents the kth facial key point in the 3D face model, 1≤k≤M, M is the total number of facial key points, M id Represents the neutral facial expression base of the target object, S_B i represents the i-th personalized facial expression base of the target object, 1≤i≤n, n is the total number of personalized facial expression bases, β i S_B i The corresponding weight coefficient, f k represents the kth facial key point in the second facial image, s represents the scale scaling factor when the three-dimensional face model is mapped to the second facial image, R represents the rigid rotation matrix when the three-dimensional face model is mapped to the second facial image, t represents the translation vector when the three-dimensional face model is mapped to the second facial image, and s, R and t are pose parameters; Alternatively, the error parameter formula is: min E exp =min||Aβ-b|| 2 , Among them, E exp represents the error parameter, β represents the weight coefficient vector, β=(β1 β2…β n ), n represents the total number of personalized facial expression bases, β i represents the weight coefficient of the personalized expression base of the i-th face of the target object, 1≤i≤n, A=sR·ΔB, s represents the scale scaling factor when the three-dimensional face model is mapped to the second face image, R represents the rigid rotation matrix when the three-dimensional face model is mapped to the second face image, ΔB=(ΔB1 ΔB2…ΔB n ), ΔB i represents the expression base parameter corresponding to the personalized expression base of the i-th face, b = ft-sR·M id , f represents the facial key points in the second face image, t represents the translation vector when the 3D face model is mapped to the second face image, s, R and t are pose parameters, M id Represents the neutral facial expression base of the target object.

9. A virtual image construction device, characterized in that: include: A first image acquisition module, used to acquire a first image acquired offline, wherein the first image includes a first face image of a target object; An expression base construction module, used to construct a neutral expression base and multiple personalized expression bases of the face of the target object according to the first face image and predetermined expression base parameters; A second image acquisition module, used to acquire a second image collected online, where the second image includes a second face image of the target object; A parameter determination module, used to determine the posture parameters required when the three-dimensional face model of the target object is mapped to the second face image and the weight coefficients of each of the personalized face expression bases according to the neutral face expression base and the plurality of personalized face expression bases; a parameter sending module, configured to send the posture parameters and the weight coefficients to a remote device, so that the remote device generates a virtual image corresponding to the second face image according to the posture parameters and the weight coefficients; The parameter determination module includes: a three-dimensional model construction unit, which is used to construct a three-dimensional face model of a target object based on a neutral face expression base and multiple personalized face expression bases; an error parameter determination unit, which is used to determine an error parameter formula when the three-dimensional face model is mapped to a second face image; a parameter calculation unit, which is used to determine, according to the error parameter formula, the posture parameters of the three-dimensional face model and the weight coefficients of each personalized face expression base when the error parameter is minimum; wherein, constructing the three-dimensional face model of the target object based on the neutral face expression base and multiple personalized face expression bases includes: using an initialization weight coefficient to linearly weight each of the personalized face expression bases and the neutral face expression base to obtain the three-dimensional face model of the target object.

10. A virtual image construction device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the virtual image construction method as described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the virtual image construction method as described in any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Virtual face animation synthesis method

    CN105528805A

  • Human face model processing method and equipment and storage medium thereof

    CN108537881A

  • Video virtual expression generation method and device, and computer readable storage medium

    CN110148191A

  • Three-dimensional facial expression base establishing method and device, storage medium and electronic equipment

    CN111768477A

  • Virtual portrait rendering method and device and storage medium

    CN111814652A