Method and device for generating virtual image rendering model
By combining neural differentiable ray tracing and traditional rendering engines, the optimization model is trained using a three-dimensional deformable face model, and the problems of high cost of virtual image generation and poor scalability in the existing technology are solved, and efficient and applicable virtual image rendering is achieved. Suitable for the generation interaction scenarios of virtual digital humans in the metaverse.
Patent Information
- Application Number
- CN202310344024.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-31
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2043-03-31
AI Technical Summary
The existing high-precision virtual image generation solution has high hardware cost, high computing cost, weak scalability, and is difficult to be applied to outdoor scenes. The traditional multi-view multi-light model is suitable for indoor scenes and is difficult to reuse.
Combining neural differentiable ray tracing neural rendering and traditional rendering engine, using a three-dimensional deformable face model and optical model prior, train and optimize the model to generate virtual avatar rendering model through face detection tracking and reconstruction, which is suitable for the meta-universe virtual digital humans. Generate interactive scenes.
It reduces computing power and hardware costs, improves terminal adaptability and rendering engine adaptability, and realizes fast and accurate virtual image rendering, which is suitable for multi-terminal virtual image generation interactive scenarios.
Smart Images

Figure CN116342782B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, augmented reality, virtual reality, deep learning and other technical fields, and specifically to a method and device for generating a virtual image rendering model. Background Art
[0002] Virtual digital humans are one of the key elements in creating the virtual world of the Metaverse. According to different business needs of digital humans, digital humans can be divided into 2D, 3D, cartoon, realistic, and hyperrealistic categories. Due to the realistic requirements of hyperrealistic digital human rendering, it is necessary to perform fine-grained modeling of digital human materials, lighting models, 3D models, etc. The hyperrealistic rendering map design of virtual images completely relies on professional designers to perform iterative optimization of the design according to business needs. Professional animators are required to professionally optimize the geometric modeling, texture mapping, lighting mapping, etc. of the virtual image to achieve the basic virtual image construction that adapts to business needs.
[0003] The existing high-precision virtual image generation solutions have high hardware costs and require the construction of multiple perspectives combined with different lighting to solve geometric textures. The computational cost is also high, and multi-perspective solutions require the establishment of high-resolution physical lighting models for each pixel to solve. In addition, the existing solutions have weak scalability, and traditional multi-perspective and multi-lighting are only applicable to indoor scenes and are difficult to reuse for outdoor unconstrained scenes. Summary of the invention
[0004] The present disclosure provides a method, apparatus, device, storage medium and computer program product for generating a virtual image rendering model.
[0005] According to a first aspect of the present disclosure, a method for generating a virtual image rendering model is provided, comprising: obtaining a video of a predetermined person; performing face detection and tracking on the video to obtain a set of face images; performing three-dimensional face reconstruction on the set of face images through a three-dimensional deformable face model to obtain expression parameters and texture parameters of each frame of the face image; training an optimization model based on the expression parameters and texture parameters of each frame of the face image; and generating a virtual image rendering model based on the three-dimensional deformable face model and the optimization model.
[0006] According to a second aspect of the present disclosure, a virtual image rendering method is provided, comprising: obtaining a target facial image to be rendered; inputting the target facial image into a virtual image rendering model trained according to any one of the methods in the first aspect to generate a 3D facial animation.
[0007] According to a third aspect of the present disclosure, there is provided a device for generating a virtual image rendering model, comprising: an acquisition unit, configured to acquire a video of a predetermined person; a tracking unit, configured to perform face detection and tracking on the video to obtain a set of face images; a reconstruction unit, configured to perform three-dimensional face reconstruction on the set of face images through a three-dimensional deformable face model to obtain expression parameters and texture parameters of each frame of the face image; an optimization unit, configured to train an optimization model based on the expression parameters and texture parameters of each frame of the face image; and a generation unit, configured to generate a virtual image rendering model based on the three-dimensional deformable face model and the optimization model.
[0008] According to a fourth aspect of the present disclosure, a virtual image rendering device is provided, comprising: an acquisition unit, configured to acquire a target facial image to be rendered; and a generation unit, configured to input the target facial image into a virtual image rendering model trained by the device according to any one of the second aspects, to generate a 3D facial animation.
[0009] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods described in the first aspect or the second aspect.
[0010] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods of the first aspect or the second aspect.
[0011] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of the first aspect or the second aspect.
[0012] According to the technology of this application, neural differentiable ray tracing neural rendering is combined with traditional rendering engines, and the geometric driving estimation of the three-dimensional implicit space of the virtual image is realized by combining the three-dimensional parameterized model and optical model prior of the virtual image design. This method has great advantages over other methods in terms of computing power cost, hardware cost, terminal adaptability, rendering engine adaptability, convergence speed, etc.
[0013] The method described in this application is not only applicable to the generation and interaction scenarios of virtual digital humans in the Metaverse, but also to the generation and interaction scenarios of virtual images in most terminals at present. This method is expected to become the standard form of generation, display and interaction of multi-terminal digital humans in the Metaverse.
[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0016] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0017] Figure 2 is a flow chart of an embodiment of a method for generating a virtual image rendering model according to the present application;
[0018] Figure 3 is a schematic diagram of an application scenario of the virtual image rendering method according to the present application;
[0019] Figure 4 is a flow chart of an embodiment of a method for rendering a virtual image according to the present application;
[0020] Figure 5 is a structural schematic diagram of an embodiment of a device for generating a virtual image rendering model according to the present application;
[0021] Figure 6 is a structural schematic diagram of an embodiment of a virtual image rendering device according to the present application;
[0022] Figure 7 It is a block diagram of an electronic device used to implement the method for generating a virtual image rendering model and the virtual image rendering method of the embodiments of the present application. DETAILED DESCRIPTION
[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0024] Figure 1 An exemplary system architecture 100 is shown in which a method for generating a virtual image rendering model, an apparatus for generating a virtual image rendering model, a virtual image rendering method or a virtual image rendering apparatus according to an embodiment of the present application can be applied.
[0025] like Figure 1As shown, the system architecture 100 may include terminals 101, 102, a network 103, a database server 104, and a server 105. The network 103 is used to provide a medium for communication links between the terminals 101, 102, the database server 104, and the server 105. The network 103 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.
[0026] The user 110 can use the terminals 101 and 102 to interact with the server 105 through the network 103 to receive or send messages, etc. Various client applications can be installed on the terminals 101 and 102, such as model training applications, video-driven face animation applications, shopping applications, payment applications, web browsers, and instant messaging tools.
[0027] The terminals 101 and 102 here can be hardware or software. When the terminals 101 and 102 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio La7er III, Moving Picture Experts Compression Standard Audio Layer 3), laptops and desktop computers, etc. When the terminals 101 and 102 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.
[0028] When the terminals 101 and 102 are hardware, a microphone and an image acquisition device may be installed thereon. The image acquisition device may be any device capable of acquiring images, such as a camera, a sensor, etc. The user 110 may use the image acquisition device on the terminals 101 and 102 to acquire faces, and use the microphone to acquire voices.
[0029] The database server 104 may be a database server that provides various services. For example, a sample set may be stored in the database server. The sample set includes a large number of samples. The samples may include a video of a predetermined person, and may also include a 3D virtual image constructed by a professional animator through geometric modeling, texture mapping, light mapping, etc. based on the image of the predetermined person. In this way, the user 110 may also select a sample from the sample set stored in the database server 104 through the terminals 101 and 102.
[0030] The server 105 may also be a server that provides various services, such as a background server that supports various applications running on the terminals 101 and 102. The background server may train the initial virtual image rendering model using samples in the sample set sent by the terminals 101 and 102, and may send the training results (such as the generated virtual image rendering model) to the terminals 101 and 102. In this way, the user may apply the generated model to drive 3D animation.
[0031] The database server 104 and the server 105 here can also be hardware or software. When they are hardware, they can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When they are software, they can be implemented as multiple software or software modules (for example, for providing distributed services), or as a single software or software module. No specific limitation is made here.
[0032] It should be noted that the method for generating a virtual image rendering model and the method for rendering a virtual image provided in the embodiment of the present application are generally executed by the server 105. Accordingly, the device for generating a virtual image rendering model and the device for rendering a virtual image are generally also disposed in the server 105.
[0033] It should be pointed out that, in the case where the server 105 can implement the related functions of the database server 104 , the database server 104 may not be provided in the system architecture 100 .
[0034] It should be understood that Figure 1 The number of terminals, networks, database servers and servers in the embodiment is only for illustration. Any number of terminals, networks, database servers and servers may be provided according to the implementation requirements.
[0035] Continue to see Figure 2 , which shows a process 200 of an embodiment of a method for generating a virtual image rendering model according to the present application. The method for generating a virtual image rendering model may include the following steps:
[0036] Step 201, obtaining a video of a predetermined person.
[0037] In this embodiment, the execution body (eg Figure 1 The server shown in the figure) can obtain the video of the predetermined person in various ways. The predetermined person can be a real person or a virtual person, such as a cartoon character. For example, the execution subject can obtain the video from the database server (such as the database server) through a wired connection or a wireless connection. Figure 1 For example, the user can obtain the video of the existing predetermined person stored in the database server 104 shown in FIG. Figure 1The terminals 101 and 102 shown in the figure are used to collect videos of predetermined characters. These videos are composed of 2D video frames.
[0038] Step 202: perform face detection and tracking on the video to obtain a set of face images.
[0039] In this embodiment, the faces in the video frames can be identified by the existing face detection technology, and then the faces can be tracked by the target tracking algorithm of the existing technology, and finally a set of face images containing the predetermined person can be extracted from the video.
[0040] Step 203, performing 3D face reconstruction on the face image set through a 3D deformable face model to obtain expression parameters and texture parameters of each frame of the face image.
[0041] In this embodiment, there are many 3D reconstruction methods based on face images, the most common of which include stereo matching, Structure From Motion (SfM for short), Shape from Shading (sfs for short), and 3D deformable face model (3DMM). 3DMM, i.e. 3D deformable face model, is a universal 3D face model that uses a fixed number of points to represent a face. Its core idea is that faces can be matched one-to-one in 3D space, and can be obtained by weighted linear addition of many other orthogonal bases of faces. In the 3D space we are in, each point (x, y, z) is actually obtained by weighted addition of base quantities in three directions of the 3D space, (1, 0, 0), (0, 1, 0), (0, 0, 1), but the weights are x, y, and z respectively.
[0042] The same principle applies to three-dimensional space. Each three-dimensional face can be represented in a basis vector space composed of all faces in a database, and solving the model of any three-dimensional face is actually equivalent to solving the coefficient problem of each basis vector.
[0043] The basic attributes of a face include shape and texture. Each face can be represented as a linear superposition of a shape vector and a texture vector. Texture parameters are linear combinations of multiple expression bases.
[0044] Each frame of the image corresponds to a set of expression parameters. The facial drive (including lip shape) of the 3D virtual portrait is controlled by a set of Blender Shape (BS, expression parameters) parameters predefined by some animation engines. The BS parameter is a vector, and each dimension ranges from [0,1], representing a specific facial movement. For example, the dimension jaw_open controls the degree of opening and closing of the jaw, and the larger the value, the greater the opening and closing. Figure 3The left image shows the virtual portrait shape corresponding to jaw_open = 0, and the right image shows the virtual portrait shape corresponding to jaw_open = 1. Similarly, a total of 370 BSs that control the various muscle dimensions of the virtual human's face jointly control the facial movement of the virtual human, allowing it to have the ability to express various expressions.
[0045] Step 204: training an optimization model based on expression parameters and texture parameters of each frame of the facial image.
[0046] In this embodiment, the optimization model can be a neural network, which can optimize at least one of the expression parameters and texture parameters, so as to obtain a hyper-realistic 3D virtual image. A professional animator can perform professional optimization design on the virtual image by geometric modeling, texture mapping, light mapping, etc. to achieve the basic virtual image construction that meets business needs, and obtain optimized expression parameters and texture parameters. The output result of step 203 is used as the input of the optimization model, and the expression parameters and texture parameters optimized by the animator are used as the output of the optimization model, and the optimization model is supervised for training.
[0047] Step 205, generating a virtual image rendering model based on the three-dimensional deformable face model and the optimization model.
[0048] In this embodiment, the three-dimensional deformable face model and the optimization model can be cascaded to form a virtual image rendering model. The three-dimensional deformable face model is used to generate a basic 3D virtual image, and then the 3D virtual image is input into the optimization model to optimize the 3D virtual image and make it more realistic.
[0049] The method provided by the above-mentioned embodiments of the present application can quickly and accurately train a virtual image rendering model, so that the 3D animation generated by the model is more realistic and achieves a hyper-realistic effect.
[0050] In some optional implementations of the present embodiment, the expression parameters and texture parameters of each frame of the facial image are used to train the optimization model, including: training a shape optimization model based on the expression parameters of each frame of the facial image as an optimization model. The shape optimization model is a neural network used to optimize the expression parameters, thereby optimizing the shape of the 3D virtual image and obtaining a 3D animation with a more realistic shape. Professional animators can perform professional optimization design on the virtual image by geometric modeling, texture mapping, light mapping, etc. to achieve a basic virtual image construction that meets business needs, and obtain optimized expression parameters and texture parameters. The expression parameters obtained in step 203 are used as the input of the shape optimization model, and the expression parameters optimized by the animator are used as the output of the shape optimization model, and the shape optimization model is supervised for training.
[0051] In some optional implementations of the present embodiment, the expression parameters and texture parameters of each frame of the face image are used to train the optimization model, including: training a color optimization model based on the texture parameters of each frame of the face image as an optimization model. The color optimization model is a neural network used to optimize the texture parameters, thereby optimizing the texture (color) of the 3D virtual image and obtaining a 3D animation with more realistic colors. Professional animators can perform professional optimization design on the virtual image by geometric modeling, texture mapping, light mapping, etc. to achieve the basic virtual image construction that meets business needs and obtain optimized texture parameters. The texture parameters obtained in step 203 are used as the input of the color optimization model, and the texture parameters optimized by the animator are used as the output of the color optimization model, and the color optimization model is supervised for training.
[0052] In some optional implementations of this embodiment, the training optimization model based on the expression parameters and texture parameters of each frame of the face image includes: training a shape optimization model based on the expression parameters of each frame of the face image; training a color optimization model based on the texture parameters of each frame of the face image; and generating an optimization model based on the shape optimization model and the color optimization model. The specific process is the same as above, and the optimization model obtained in this way includes a shape optimization model and a color optimization model, thereby achieving color optimization and shape optimization of 3D animation, and obtaining a 3D animation with more realistic shape and color.
[0053] In some optional implementations of this embodiment, the shape optimization model is trained based on the expression parameters of each frame of the face image as the optimization model, including: mapping and weighting the expression parameters of each frame of the face image through the shape optimization model to obtain the weighted result of the expression parameters of each frame of the face image. The shape optimization model is a three-dimensional motion voxel grid base or a signed distance field. The three-dimensional motion voxel grid is a three-dimensional grid map with a fixed resolution, which is used for voxel rendering. What is rendered is not a single face, but a solid object. The signed distance field (Signed Distance Field, referred to as SDF) can be represented by a scalar field function or a volume map. Simply put, it is a spatial expression, which stores the distance from a point in space to the nearest triangular face, and if it is inside the object, it is a negative value. Because it is a scalar field of "distance", the use of "distance" retains spatial information, and combined with hardware interpolation, the surface information can be better retained. Unlike voxels, once voxelized, all geometric information is lost, and the original boundary is unknown, so the effective resolution of SDF is actually much higher than that of voxels. SDF can speed up when tracking and can jump over empty spaces.
[0054] In some optional implementations of this embodiment, the shape optimization model is trained based on the expression parameters of each frame of the face image as the optimization model, including: realizing the calibration space learning of the expression parameters of each frame of the face image through the shape optimization model. Since the parameterized basis has the problem of insufficient expressiveness, calibration is required. The shape optimization model can be a multi-layer perceptron. The expression parameters of each frame of the face image can be used as the input of the shape optimization model, and the expression parameters after manual calibration can be used as the output of the shape optimization model, and the shape optimization model is supervisedly trained. A model that can convert the expression parameters into optimized expression parameters is obtained.
[0055] In some optional implementations of this embodiment, the shape optimization model is trained based on the expression parameters of each frame of the face image as the optimization model, including: mapping and weighting the expression parameters of each frame of the face image through the three-dimensional motion voxel grid basis in the shape optimization model to obtain the weighted result of the expression parameters of each frame of the face image; based on the weighted result of the expression parameters of each frame of the face image, the calibration space learning of the weighted result of the expression parameters of each frame of the face image is realized through the multi-layer perceptron in the shape optimization model. The offset can be corrected by mapping weighting first, and then further corrected by calibration, so as to obtain a 3D face with a more accurate shape.
[0056] In some optional implementations of this embodiment, the color optimization model is trained based on the texture parameters of each frame of the face image as the optimization model, including: training a multi-layer perceptron as the optimization model based on the texture parameters of each frame of the face image. The texture parameters of each frame of the face image can be used as the input of the color optimization model, and the texture parameters after manual calibration can be used as the output of the color optimization model to train the color optimization model in a supervised manner. A model that can convert texture parameters into optimized texture parameters is obtained.
[0057] In some optional implementations of this embodiment, the color optimization model based on the texture parameters of each frame of face image is used as the optimization model, including: mapping and weighting the expression parameters of each frame of face image through the three-dimensional motion voxel grid basis in the shape optimization model based on the expression parameters of each frame of face image, and obtaining the weighted result of the expression parameters of each frame of face image; based on the weighted result of the expression parameters of each frame of face image, the calibration space learning of the weighted result of the expression parameters of each frame of face image is realized through the multi-layer perceptron in the shape optimization model, and the calibration result of the expression parameters of each frame of face image is obtained; based on the calibration result of the expression parameters of each frame of face image and the texture parameters of each frame of face image, the color optimization model is trained as the optimization model. The training of the shape optimization model and the training of the texture optimization model can be combined to train an optimization model that optimizes both shape and texture.
[0058] In some optional implementations of this embodiment, the method further includes: performing self-supervisory training on the virtual image rendering model based on the expression parameters and texture parameters of each frame of the face image. After the virtual image rendering model training is completed, the expression parameters and texture parameters regressed in step 203 can be used to realistically drive the implicit space three-dimensional virtual image to obtain a 3D animation. The 3D animation can be projected and converted into a 2D image, and the loss value between the converted 2D image and the face image obtained in step 202 is calculated. The virtual image rendering model is adjusted by the loss value to achieve self-supervised training.
[0059] Continue to see Figure 4 , which shows a process 400 of an embodiment of a virtual image rendering method provided by the present application. The virtual image rendering method may include the following steps:
[0060] Step 401, obtaining a target face image to be rendered.
[0061] In this embodiment, the execution subject of the virtual image rendering method (for example Figure 1 The server 105 shown in FIG. 104 may obtain the target face image in a variety of ways. For example, the execution subject may obtain the target face image from a database server (e.g. Figure 1 For another example, the execution subject may also receive a terminal (such as Figure 1 The target face image is collected by the terminals 101, 102) or other devices shown.
[0062] Step 402, input the target face image into the virtual image rendering model to generate a 3D face animation.
[0063] In this embodiment, the target face image is input into the virtual image rendering model trained according to the method described in steps 201-205, expression parameters and texture parameters can be extracted, and then the expression parameters and texture parameters are optimized to obtain an optimized 3D face animation.
[0064] It should be noted that the virtual image rendering method of this embodiment can be used to test the virtual image rendering model generated by the above embodiments. Then, the virtual image rendering model can be continuously optimized according to the test results. This method can also be a practical application method of the virtual image rendering model generated by the above embodiments. Using the virtual image rendering model generated by the above embodiments to generate a 3D face animation helps to improve the realism of the 3D face animation.
[0065] The method described in this application is not only applicable to the generation and interaction scenarios of virtual digital humans in the Metaverse, but also to the generation and interaction scenarios of virtual images in most terminals at present. This application proposes a virtual image neural rendering driving method, which is expected to become the standard form of multi-terminal digital human generation and display interaction in the Metaverse.
[0066] Continue to see Figure 5 , as a response to the above Figure 2 The present application provides an embodiment of a device for generating a virtual image rendering model. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0067] like Figure 5 As shown, the apparatus 500 for generating a virtual image rendering model in this embodiment may include: an acquisition unit 501, a tracking unit 502, a reconstruction unit 503, an optimization unit 504, and a generation unit 505. The acquisition unit 501 is configured to acquire a video of a predetermined person; the tracking unit 502 is configured to perform face detection and tracking on the video to obtain a set of face images; the reconstruction unit 503 is configured to perform three-dimensional face reconstruction on the set of face images through a three-dimensional deformable face model to obtain expression parameters and texture parameters of each frame of face image; the optimization unit 504 is configured to train an optimization model based on the expression parameters and texture parameters of each frame of face image; and the generation unit 505 is configured to generate a virtual image rendering model according to the three-dimensional deformable face model and the optimization model.
[0068] In some optional implementations of this embodiment, the optimization unit 504 is further configured to: train a shape optimization model based on expression parameters of each frame of facial image as the optimization model.
[0069] In some optional implementations of this embodiment, the optimization unit 504 is further configured to: train a color optimization model based on texture parameters of each frame of the face image as the optimization model.
[0070] In some optional implementations of this embodiment, the optimization unit 504 is further configured to: train a shape optimization model based on expression parameters of each frame of facial image; train a color optimization model based on texture parameters of each frame of facial image; and generate an optimization model based on the shape optimization model and the color optimization model.
[0071] In some optional implementations of this embodiment, the optimization unit 504 is further configured to: map and weight the expression parameters of each frame of the face image through the shape optimization model to obtain a weighted result of the expression parameters of each frame of the face image.
[0072] In some optional implementations of this embodiment, the shape optimization model is a three-dimensional motion voxel grid basis.
[0073] In some optional implementations of this embodiment, the shape optimization model is a signed distance field.
[0074] In some optional implementations of this embodiment, the optimization unit 504 is further configured to: implement calibration space learning of expression parameters of each frame of facial image through a shape optimization model.
[0075] In some optional implementations of the present embodiment, the optimization unit 504 is further configured to: map and weight the expression parameters of each frame of facial image through the three-dimensional motion voxel grid basis in the shape optimization model to obtain the weighted results of the expression parameters of each frame of facial image; based on the weighted results of the expression parameters of each frame of facial image, implement calibration space learning of the weighted results of the expression parameters of each frame of facial image through the multi-layer perceptron in the shape optimization model.
[0076] In some optional implementations of this embodiment, the optimization unit 504 is further configured to: train a multi-layer perceptron as an optimization model based on texture parameters of each frame of the face image.
[0077] In some optional implementations of the present embodiment, the optimization unit 504 is further configured to: map and weight the expression parameters of each frame of facial image through the three-dimensional motion voxel grid basis in the shape optimization model to obtain the weighted results of the expression parameters of each frame of facial image; implement calibration space learning of the weighted results of the expression parameters of each frame of facial image through the multi-layer perceptron in the shape optimization model to obtain the calibration results of the expression parameters of each frame of facial image; and train a color optimization model based on the calibration results of the expression parameters of each frame of facial image and the texture parameters of each frame of facial image as an optimization model.
[0078] In some optional implementations of this embodiment, the device 500 also includes a training unit (not shown in the drawings), which is configured to perform self-supervised training on the virtual image rendering model based on expression parameters and texture parameters of each frame of the facial image.
[0079] Continue to see Figure 6 , as a response to the above Figure 4 The present application provides an embodiment of a virtual image rendering device. Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0080] like Figure 6As shown, the virtual image rendering device 600 of this embodiment may include: an acquisition unit 601 and a generation unit 602, wherein the acquisition unit 601 is configured to acquire a target face image to be rendered; and the generation unit 602 is configured to input the target face image into the virtual image rendering model trained by the device 500 to generate a 3D face animation.
[0081] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0082] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0083] An electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in process 200 or 400.
[0084] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method described in process 200 or 400.
[0085] A computer program product includes a computer program, wherein the computer program implements the method described in flow 200 or 400 when executed by a processor.
[0086] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0087] like Figure 7As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0088] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0089] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a method for generating a virtual image rendering model. For example, in some embodiments, the method for generating a virtual image rendering model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the method for generating a virtual image rendering model described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the method for generating a virtual image rendering model in any other appropriate manner (e.g., by means of firmware).
[0090] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0091] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0092] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0093] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0094] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0095] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0096] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0097] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method for generating a virtual image rendering model, comprising: Get the video of the intended person; Performing face detection and tracking on the video to obtain a face image set; Performing 3D face reconstruction on the face image set through a 3D deformable face model to obtain expression parameters and texture parameters of each frame of the face image; Training an optimization model based on at least one of an expression parameter and a texture parameter of each frame of the face image, including: training a color optimization model as an optimization model based on the texture parameter of each frame of the face image; Generate a virtual image rendering model according to the three-dimensional deformable face model and the optimization model; The color optimization model is trained based on the texture parameters of each frame of the face image as the optimization model, including: Based on the expression parameters of each frame of the face image, the expression parameters of each frame of the face image are mapped and weighted through a three-dimensional motion voxel grid basis in the shape optimization model to obtain a weighted result of the expression parameters of each frame of the face image; Based on the weighted results of the expression parameters of each frame of the face image, the calibration space learning of the weighted results of the expression parameters of each frame of the face image is realized through the multi-layer perceptron in the shape optimization model, so as to obtain the calibration results of the expression parameters of each frame of the face image; A color optimization model is trained based on the calibration results of the expression parameters of each frame of the face image and the texture parameters of each frame of the face image as the optimization model.
2. The method according to claim 1, wherein: The training optimization model based on at least one of the expression parameter and the texture parameter of each frame of the face image also includes: The shape optimization model is trained based on the expression parameters of each frame of the face image as the optimization model.
3. The method according to claim 1, wherein: The training optimization model based on at least one of the expression parameter and the texture parameter of each frame of the face image also includes: The shape optimization model is trained based on the expression parameters of each face image frame; The color optimization model is trained based on the texture parameters of each face image frame; An optimization model is generated according to the shape optimization model and the color optimization model.
4. The method according to claim 2, wherein: The shape optimization model is trained based on expression parameters of each frame of facial image as an optimization model, including: The expression parameters of each frame of the face image are mapped and weighted through the shape optimization model to obtain the weighted result of the expression parameters of each frame of the face image.
5. The method according to claim 4, wherein: The shape optimization model is a three-dimensional motion voxel grid basis.
6. The method according to claim 4, wherein: The shape optimization model is a signed distance field.
7. The method according to claim 2, wherein: The shape optimization model is trained based on expression parameters of each frame of facial image as an optimization model, including: The shape optimization model is used to achieve calibration space learning of expression parameters of each frame of facial image.
8. The method according to claim 2, wherein: The shape optimization model is trained based on expression parameters of each frame of facial image as an optimization model, including: The expression parameters of each frame of the face image are mapped and weighted by using the three-dimensional motion voxel grid basis in the shape optimization model to obtain a weighted result of the expression parameters of each frame of the face image; Based on the weighted results of the expression parameters of each frame of the face image, the calibration space learning of the weighted results of the expression parameters of each frame of the face image is realized through the multi-layer perceptron in the shape optimization model.
9. The method according to claim 1, wherein: The color optimization model trained based on the texture parameters of each frame of the face image is used as the optimization model, and further includes: A multi-layer perceptron is trained as the optimization model based on the texture parameters of each frame of the face image.
10. The method according to any one of claims 1 to 9, wherein: The method further comprises: The virtual image rendering model is self-supervised trained based on the expression parameters and texture parameters of each frame of the facial image.
11. A method for rendering a virtual image, comprising: Get the target face image to be rendered; Input the target face image A virtual image rendering model trained according to any one of the methods of claims 1-10 generates a 3D facial animation.
12. A device for generating a virtual image rendering model, comprising: An acquisition unit, configured to acquire a video of a predetermined person; A tracking unit, configured to perform face detection and tracking on the video to obtain a set of face images; A reconstruction unit is configured to perform three-dimensional face reconstruction on the face image set through a three-dimensional deformable face model to obtain expression parameters and texture parameters of each frame of the face image; The optimization unit is configured to train an optimization model based on at least one of an expression parameter and a texture parameter of each frame of the face image, including: training a color optimization model as an optimization model based on the texture parameter of each frame of the face image; A generating unit, configured to generate a virtual image rendering model according to the three-dimensional deformable face model and the optimization model; Wherein, the optimization unit is further configured to: The expression parameters of each frame of the face image are mapped and weighted by using the three-dimensional motion voxel grid basis in the shape optimization model to obtain a weighted result of the expression parameters of each frame of the face image; The calibration space learning of the weighted results of the expression parameters of each frame of the face image is realized by the multi-layer perceptron in the shape optimization model, so as to obtain the calibration results of the expression parameters of each frame of the face image; A color optimization model is trained based on the calibration results of the expression parameters of each frame of the face image and the texture parameters of each frame of the face image as the optimization model.
13. The device according to claim 12, wherein: The optimization unit is further configured to: The shape optimization model is trained based on the expression parameters of each frame of the face image as the optimization model.
14. The device according to claim 12, wherein: The optimization unit is further configured to: The shape optimization model is trained based on the expression parameters of each face image frame; The color optimization model is trained based on the texture parameters of each face image frame; An optimization model is generated according to the shape optimization model and the color optimization model.
15. The device according to claim 13, wherein: The optimization unit is further configured to: Based on the expression parameters of each frame of the face image, the expression parameters of each frame of the face image are mapped and weighted through a shape optimization model to obtain a weighted result of the expression parameters of each frame of the face image.
16. The device according to claim 15, wherein: The shape optimization model is a three-dimensional motion voxel grid basis.
17. The device according to claim 15, wherein: The shape optimization model is a signed distance field.
18. The device according to claim 13, wherein: The optimization unit is further configured to: The shape optimization model is used to achieve calibration space learning of expression parameters of each frame of facial image.
19. The device according to claim 13, wherein: The optimization unit is further configured to: The expression parameters of each frame of the face image are mapped and weighted by using the three-dimensional motion voxel grid basis in the shape optimization model to obtain a weighted result of the expression parameters of each frame of the face image; Based on the weighted results of the expression parameters of each frame of the face image, the calibration space learning of the weighted results of the expression parameters of each frame of the face image is realized through the multi-layer perceptron in the shape optimization model.
20. The device according to claim 12, wherein: The optimization unit is further configured to: A multi-layer perceptron is trained as the optimization model based on the texture parameters of each frame of the face image.
21. The device according to any one of claims 12 to 20, wherein: The apparatus further comprises a training unit configured to: The virtual image rendering model is self-supervised trained based on the expression parameters and texture parameters of each frame of the facial image.
22. A virtual image rendering device, comprising: An acquisition unit, configured to acquire a target face image to be rendered; A generating unit is configured to input the target face image into A virtual image rendering model trained by the device according to any one of claims 12-21 generates a 3D facial animation.
23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.
24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.
25. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.
