Method and device for generating virtual characters

Abstract

The present application relates to the technical field of three-dimensional reconstruction, and provides a method and device for generating a virtual character. A video of a target object rotating in a preset posture is collected by a civilian-grade RGB camera. Since the front image of the target object is rich in detail features, it can be used as the first video frame to extract the body parameters of the target object to voxelize the initial human body parameterized model, generate a reference frame model with skin, and continuously optimize the reference frame model through the posture parameters extracted from each video frame to obtain a target geometric model. Then, after texture mapping of the target geometric model according to the RGB information of each video frame, a virtual character consistent with the appearance of the target object is obtained. Through continuous optimization, the authenticity of the virtual character is improved. In the optimization process, the loss of the reference frame model before and after the deformation of the previous video frame is used to optimize the deformation parameters of the next video frame. There is no need to manually adjust the model, which improves the generation efficiency of the virtual character.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional reconstruction, and in particular to a method and device for generating a virtual character. Background Art

[0002] With the development of three-dimensional reconstruction technology, in virtual space, realistic virtual characters can interact with other virtual characters in real time and remotely in three dimensions, recreating the face-to-face immersion in the first generation of social methods. Virtual / augmented reality may become the fifth generation of social methods after the fourth generation of mobile Internet era.

[0003] At present, there are two main ways to implement remote social interaction based on human body 3D reconstruction: one is to perform 3D reconstruction of the human body based on real-time collected RGB data or RGBD data to obtain a virtual character, and transmit the 3D data of the virtual character (such as vertex coordinates, patch index, texture, etc.) to other interactive terminals through the cloud; since this method is based on the reconstruction of the real data collected, the human body, clothes, hair, etc. of the virtual character are more realistic, but the amount of transmission is large, and the existing network bandwidth is difficult to meet the real-time requirements, resulting in the jamming of the interaction process and reducing the user's immersive experience. The other is to pre-build a virtual character, and then use motion capture technology to collect the user's motion data in real time during the interaction process to drive the virtual character to move; this method reduces the amount of data transmitted during the real-time interaction, but when this method is reconstructed using civilian-grade acquisition equipment, it can only interact in real time when driving a virtual character without clothes or wearing preset clothes. If the clothes of the virtual character are required to be similar to the user and move with the user, they can only be manually drawn by animators using film-level methods, and real-time performance cannot be achieved. Summary of the invention

[0004] The embodiments of the present application provide a method and device for generating a virtual character, so as to improve the authenticity and generation efficiency of the virtual character while satisfying the condition of real-time interaction.

[0005] On the one hand, an embodiment of the present application provides a method for generating a virtual character, comprising:

[0006] Obtain a video captured by an RGB camera of a target object rotating in a preset posture, and use the front image of the target object as the first video frame to extract the shape parameters of the target object;

[0007] Initializing a voxel space of a set resolution according to the initial human body parameterized model and the body parameters, and generating a reference frame model with skin according to the initialization result;

[0008] According to the posture parameters of the target object contained in each video frame and the three-dimensional coordinates of each vertex in the reference frame model in the corresponding voxel block, the reference frame model is deformed to obtain the target geometric model; during the deformation process, the loss of the reference frame model corresponding to the previous video frame before and after deformation is used to optimize the deformation parameters of the next video frame;

[0009] The target geometric model is texture mapped according to the RGB information of each video frame to obtain a virtual character of the target object.

[0010] On the other hand, an embodiment of the present application provides an interactive device for generating a virtual character for remote interaction, the interactive device comprising a processor, a memory, a display and a communication interface, the communication interface, the display, the memory and the processor being connected via a bus;

[0011] The memory includes a data storage unit and a program storage unit, the program storage unit stores computer program instructions, and the processor executes the following operations according to the computer program:

[0012] Through the communication interface, a video of the target object rotating in a preset posture captured by the RGB camera is obtained and stored in the data storage unit, and the front image of the target object is used as the first video frame to extract the body parameters of the target object;

[0013] Initializing a voxel space of a set resolution according to an initial human body parameterized model and the body parameters of the target object, and generating a reference frame model with skin according to the initialization result;

[0014] According to the posture parameters of the target object contained in each video frame and the three-dimensional coordinates of each vertex in the reference frame model in the corresponding voxel block, the reference frame model is deformed to obtain the target geometric model; during the deformation process, the loss of the reference frame model corresponding to the previous video frame before and after deformation is used to optimize the deformation parameters of the next video frame;

[0015] The target geometric model is texture mapped according to the RGB information of each video frame to obtain a virtual character of the target object, which is then displayed on the display.

[0016] On the other hand, the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method for generating a virtual character provided in an embodiment of the present application.

[0017] The embodiment of the present application provides a method and device for generating a virtual character, using a civilian-grade RGB camera to collect and store a video of a target object rotating in a preset posture, taking a front image with rich detail information as the first video frame to extract the body parameters of the target object, and using the body parameters to voxelize the initial human body parameterized model with consistent topological structure, and then generate a reference frame model with skin; after obtaining the reference frame model with skin, according to the posture parameters contained in each video frame, and the three-dimensional coordinates of each vertex in the reference frame model in the corresponding voxel block, respectively deform the reference frame model to obtain a target geometric model, and during the deformation process, the loss of the reference frame model before and after deformation corresponding to the previous video frame is used to optimize the deformation parameters of the next video frame, so that the target geometric model is more consistent with the target object, and the authenticity of the target geometric model is improved, so that after texture mapping of the target geometric model according to the RGB information of each video frame, a more realistic virtual character is obtained, and the skin of the virtual character can truly depict the appearance of the target object. During the entire generation process, the optimization of the deformation parameters is automatically performed without the participation of animators, saving manpower and material resources, and improving the generation efficiency of virtual characters. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1A A schematic diagram of an application scenario provided for an embodiment of the present application;

[0020] Figure 1B A schematic diagram of another application scenario provided for an embodiment of the present application;

[0021] Figure 2 A system framework diagram for generating a virtual character provided in an embodiment of the present application;

[0022] Figure 3 A flowchart of a method for generating a virtual character provided in an embodiment of the present application;

[0023] Figure 4 A flow chart of a method for extracting body parameters and posture parameters provided in an embodiment of the present application;

[0024] Figure 5 A flow chart of a voxel processing method provided in an embodiment of the present application;

[0025] Figure 6 A flow chart of a method for generating a reference frame model provided in an embodiment of the present application;

[0026] Figure 7 A flow chart of a method for deforming a reference frame model using current posture parameters provided in an embodiment of the present application;

[0027] Figure 8 A flow chart of a method for optimizing deformation parameters of a next video frame based on the loss of a previous video frame before and after deformation provided in an embodiment of the present application;

[0028] Fig. 9 The effect diagram of the target geometric model with skin provided in the embodiment of the present application;

[0029] Fig.10 A flow chart of a method for determining the correspondence between model vertices and pixel points of a texture map provided in an embodiment of the present application;

[0030] Fig.11 A flow chart of a method for texture mapping of a target geometric model provided in an embodiment of the present application;

[0031] Fig.12 The rendering of the generated virtual character provided in the embodiment of the present application;

[0032] Fig.13 A flow chart of a method for remote interaction using a virtual character provided in an embodiment of the present application;

[0033] Fig.14A A structural diagram of an interactive device provided in an embodiment of the present application;

[0034] Fig. 14B A structural diagram of another interactive device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] At present, the core technology of remote 3D interactive system is real-time 3D reconstruction of human body, face, hair and clothes. 3D reconstruction involves data such as shape, motion and material. First, the data of the object to be reconstructed is obtained from various acquisition devices as input parameters, and then the input parameters are processed by non-rigid real-time 3D reconstruction method to obtain 3D information of human body, so as to reconstruct the virtual character of the object to be reconstructed, so as to improve the novelty and fun of interaction.

[0037] According to the different sensor types, acquisition equipment can be divided into optical scanners (such as visible structured light or laser scanners), RGBD cameras, and RGB cameras. Among them, the virtual character generation method based on optical scanners is often used in scenes with high requirements for authenticity (such as film and television). By scanning the corresponding characters to obtain 3D point cloud data, and reorganizing the 3D point cloud data into human body mesh data, after careful polishing and modification by animators, high-quality virtual characters are obtained.

[0038] In recent years, with the continuous development of imaging technology, the method of generating virtual characters based on RGBD cameras has made rapid progress. This method uses Simultaneous Localization and Mapping (SLAM) technology to perform three-dimensional reconstruction, which improves the efficiency of generating virtual characters. In order to quickly generate virtual characters and reduce reconstruction costs, the method of generating virtual characters based on civilian-grade RGB cameras has become a research hotspot.

[0039] Among the products currently on the market, some generate realistic or cartoon virtual characters through single-frame 2D images, but the simulation effect is poor due to the small amount of model data; some use a large number of camera arrays for 3D data collection, and use greater computing power for modeling to generate virtual characters with better simulation effects, but require a wider network bandwidth and higher system costs; others are equipped with depth sensors to collect depth data of characters, combine RGB data for motion capture, extract motion data for driving three-dimensional models, and then drive pre-built models to obtain virtual characters.

[0040] At present, among various methods for generating virtual characters, the reconstruction method based on optical scanners has the highest accuracy. This is because during the scanning process, the object to be reconstructed is required to remain still for several seconds or minutes. By splicing high-precision 3D scanning data from multiple angles, a high-precision static human 3D model is reconstructed. Then, animators perform targeted manual repairs and embed skeleton information to convert the static human 3D model into a drivable virtual character with clothes. Since this method requires manual participation, the efficiency of generating virtual characters is low.

[0041] 3D reconstruction technology based on real-time data acquisition often fails to generate a 3D model with consistent topological structure, cannot guarantee the closure of the model, and cannot guarantee the quality of reconstruction. In addition, there is no corresponding mature compression solution for the transmission of 3D data. Under the transmission capacity of the wide area network, a large amount of 3D data cannot be transmitted in real time, which makes this method unable to be widely used. Generally, it can only be used for two-way 3D data transmission between one person and two people in a dedicated local area network. Therefore, this method does not have the feasibility of remote 3D interaction.

[0042] The method of driving pre-modeling to generate virtual characters can complete remote social interaction by transmitting motion data with a small amount of data. The generation method is simple and can achieve real-time transmission under the existing network capabilities. However, due to the human body parametric model with consistent topological structure used in the pre-modeling process, it is difficult to depict the real appearance of the character, and it is difficult to directly generate the character's clothes and hair. Real-time interaction is only possible when driving a virtual character without clothes or wearing pre-set clothes. If the appearance of the virtual character needs to be consistent with the real appearance, the animator needs to fill it in manually later, and different clothes need to be repetitive manual work. If the clothes of the virtual character are required to be similar to the user and move with the user, the animator needs to repeatedly drive the simulation and manually optimize, which is inefficient.

[0043] In view of this, the embodiment of the present application provides a method and device for generating a virtual character, which uses a video of a target object rotating in a preset posture collected by a civilian-grade RGB camera to generate a virtual character with skin (such as clothes, hair, hat, etc.) consistent with the target object. During the generation process, the initial human body parameterized model is voxelized using the body parameters of the target object contained in the first video frame, and a reference frame model with skin is generated. Based on the coordinates of each vertex of the reference frame model after voxelization and the posture parameters of the target object contained in each video frame, the reference frame model is continuously optimized to improve the authenticity of the virtual character. In the optimization process, the optimization loss of the previous video frame is used to adjust the optimization result of the next video frame, further improving the authenticity of the virtual character. At the same time, using the RGB information in each video frame, real and complete texture data can be obtained, so that the virtual character after mapping is consistent with the target object. The entire optimization process does not require the participation of an animator, saving manpower and material resources and improving the efficiency of generating virtual characters.

[0044] After obtaining the virtual character of the target object, during the remote three-dimensional interaction process, motion data used to drive the movement of the virtual character needs to be transmitted, so that a virtual character with the same movements as the target object can be obtained, reducing the amount of data transmitted over the network. Under the conditions of the existing network bandwidth, it meets the interaction requirements and improves the user's immersive experience.

[0045] See also Figure 1A , is a schematic diagram of an application scenario provided by an embodiment of the present application, in which user A and user B use a smart phone 101 and a laptop computer 102 as interactive devices to interact with each other. During the interaction, the three-dimensional model data of the two are uploaded to the server 200 through the same local area network, and the interactive device downloads the three-dimensional model data of the other party from the server for rendering and display, thereby placing the virtual characters of the two in the same virtual space, realizing an immersive experience of face-to-face interaction.

[0046] During the interaction process, interactive devices in the same LAN can also remotely interact with interactive devices in other LANs. Figure 1B , the smart phone 101 and the smart TV 103 are in the local area network 1, and the laptop computer 102 is in the local area network 2. The smart phone 101 and the smart TV 103 upload their own three-dimensional model data to the server 200 through the local area network 1, and the laptop computer uploads its own three-dimensional model data to the server 200 through the local area network 2.

[0047] It should be noted that Figure 1A and Figure 1B This is only an example. The embodiments of the present application do not impose any restrictive requirements on the interactive terminal. In addition to smart phones, smart TVs and laptops, it can also be interactive devices such as tablets, desktops and VR head-mounted display devices. Among them, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0048] Based on the above application scenarios, Figure 2 This is a system framework diagram for generating a virtual character provided in an embodiment of the present application. The system includes a civilian-grade RGB camera and a host computer equipped with a high-performance graphics processing unit (GPU), which can be divided into a data acquisition module, a geometry reconstruction module, a texture reconstruction module and a virtual character generation module.

[0049] The data acquisition module includes a civilian-grade RGB camera, which is used to collect images of the target object. During the acquisition process, the target object first faces the RGB camera, makes a Tpose or Apose gesture, and then slowly rotates one circle. During the rotation, the target object can move, and the target object's clothes and hair can also move. After the rotation is completed, the video data collected by the RGB camera is stored.

[0050] The geometric reconstruction module uses a human body parameterized model (human body geometric mesh) corresponding to the target object to initialize the fixed-resolution voxel space to obtain the reference frame model, and then uses two deep learning networks to learn the transformation relationship from each stored video frame to the reference frame model, and continuously optimizes the non-rigid deformation relationship and skinning weights to enrich the details of the reference frame model, thereby obtaining a target geometric model with consistent topological structure, thereby improving the authenticity of the virtual character.

[0051] The texture reconstruction module establishes the correspondence between the RGB information in each frame of video and the target geometric model to complete the texture of the reference frame (the first video frame in the embodiment of the present application) and obtain a complete texture map of the target geometric model.

[0052] The virtual character generation module uses the complete texture map to perform texture mapping on the target geometric model to obtain a virtual character with the same skin and posture as the target object.

[0053] In an embodiment of the present application, by continuously optimizing the reference frame model, the skin weights that drive the movement of the target geometric model of the human body can be extracted. In this way, during the remote interaction process, after obtaining the motion data of the human body parametric model from the real-time collected RGB image, the skin weights are combined to drive the movement of the target geometric model, thereby obtaining a virtual character that matches the current action of the target object. The motion data transmitted during the entire interaction process occupies a small amount of data, and under the condition of the existing network agent width, it can meet the real-time interaction needs.

[0054] See also Figure 3 , a flow chart of a method for generating a virtual character provided in an embodiment of the present application, the process is executed by an interactive device equipped with a high-performance GPU, and mainly includes the following steps:

[0055] S301: Obtain a video captured by an RGB camera showing a target object rotating in a preset posture, and use a front image of the target object as the first video frame to extract body parameters of the target object.

[0056] The RGB camera may be a camera configured on the interactive device, or may be a camera independent of the interactive device.

[0057] In the specific implementation, the RGB camera is fixed, and the target object walks in front of the RGB camera, facing the RGB camera, and the position of the target object ensures that the field of view of the RGB camera can capture the full body data of the target object. Then, the RGB camera is started, and the target object rotates in a preset posture (such as T pose or A pose) within the field of view of the RGB camera. The RGB camera captures the target object from multiple angles, generates a video of the target object, and stores it. The video can be transmitted to the GPU of the interactive device for processing through the communication interface.

[0058] Among them, the front image of the target object contains relatively rich detail information, so the captured front image of the target object is used as the first video frame, also called a reference frame.

[0059] In an embodiment of the present application, in order to generate a high-quality virtual character, the resolution of the RGB camera may be set to be greater than 720p.

[0060] It should be noted that the embodiment of the present application does not impose any restrictive requirements on the rotation method of the target object, such as standing on a mechanical turntable and rotating in a fixed posture. In order to ensure the integrity of the virtual character and the authenticity of subsequent driving, the target object can rotate with a certain amplitude during the rotation process.

[0061] S302: Initializing a voxel space of a set resolution according to an initial human body parameterized model and body parameters of a target object, and generating a reference frame model with skin according to the initialization result, wherein the body parameters are extracted from the first video frame.

[0062] After obtaining the video of the target object, each video frame is fitted with a preset initial human body parameterized model to determine the body parameters of the target object and the posture parameters of the target object contained in each video frame. The initial human body parameterized model can be an SMPL model, and its expression formula is as follows:

[0063] M(β,θ)=W(T p (β, θ), J(β), θ, ω) Formula 1

[0064] T p (β, θ) = T + B s (β)+B p (θ) Formula 2

[0065] Among them, M(β,θ) is the SMPL model construction function, T p (β, θ) is the mixed deformation function for human posture correction, β is the body parameter, θ is the posture parameter, W is the skinning function, B s (β) is a linear function constructed by body parameters of people with different body types, B p (θ) is the influence function of posture parameters of people with different body shapes on human body shape, T is the average body parameter of different body parameters, J(β) is the function for predicting the position of different human joint points, and ω is the mixed weight.

[0066] For the detailed fitting process, see Figure 4 , mainly includes the following steps:

[0067] S3020_1: For the first video frame, extract the target skeleton and the first human body contour of the target object.

[0068] When executing S3020_1, a 2D human skeleton extraction algorithm is used to extract the target skeleton of the target object from the first video frame, and a portrait segmentation algorithm is used to extract the human region of the target object from the first video frame to obtain a clear first human contour of the target object.

[0069] S3020_2: Initialize the shape parameters and posture parameters of the target skeleton, and deform the initial human body parameterized model through the skinning function.

[0070] When executing S3020_2, a set of body parameters and posture parameters are estimated based on the extracted target skeleton, the estimated values ​​are used as the initialized body parameters and posture parameters, and the initial human body parameterized model is deformed through the skinning function.

[0071] S3020_3: Extract the reference skeleton of the deformed human body parameterized model, and determine the skeleton loss value between the reference skeleton and the target skeleton.

[0072] Among them, the reference skeleton extracted from the deformed human body parameterized model is the 3D skeleton, and the skeleton loss value between it and the 2D target skeleton is calculated, which is denoted as lossl.

[0073] S3020_4: According to the intrinsic parameters of the RGB camera, the reference skeleton is projected into the first video frame to optimize the skeleton loss value, and the extrinsic parameters of the RGB camera and the new posture parameters are obtained.

[0074] Generally, the intrinsic parameters of the RGB camera are known before leaving the factory. In order to improve the accuracy of solving the shape parameters and posture parameters, the intrinsic parameters of the RGB camera can be determined by pre-calibration.

[0075] When executing S3020_4, the reference skeleton is projected into the first video frame according to the intrinsic parameters of the RGB camera, so that the reference skeleton is consistent with the target skeleton to optimize (reduce) loss1, and new posture parameters and extrinsic parameters of the RGB camera are obtained.

[0076] S3020_5: According to the new posture parameters and external parameters, the vertices of the deformed human body parameterized model are projected into the first video frame to obtain a second human body contour.

[0077] When executing S3020_5, the external parameters of the RGB camera are fixed, the new posture parameters are used as initial values, the vertices of the deformed human body parameterized model are projected into the first video frame, and the second human body contour corresponding to the human body parameterized model is obtained.

[0078] S3020_6: Update body parameters and posture parameters according to the contour loss values ​​of the first human body contour and the second human body contour.

[0079] The contour loss value of the first human contour and the second human contour is recorded as loss2. By reducing loss2, the body parameters and posture parameters of the target object contained in the first video frame are updated, and the updated body parameters are used as the final body parameters of the target object, and the updated posture parameters are used as the posture parameters of the target object contained in the first video frame.

[0080] S3020_7: Fix the external parameters of the RGB camera and the body parameters of the target object, and for non-first video frames, re-determine the skeleton loss value and the contour loss value to determine the posture parameters corresponding to the non-first video frames.

[0081] When executing S3020_7, for non-first video frames, the process of determining the skeleton loss value loos1 and the contour loss value loss2 is the same as that of the first video frame and will not be repeated here. The posture parameters of the target object contained in the non-first video frame can be obtained through the skeleton loss value loos1 and the contour loss value loss2.

[0082] After obtaining the body parameters of the target object, when executing S302, the initial human body parameterized model is voxelized according to the voxel space of the set resolution (such as 512*512*512 or higher), the human body parameterized model of T pose or A pose is mapped to the voxel space, and based on the voxelized result, a reference frame model with skin is generated. For the specific process, see Figure 5 :

[0083] S3021: deforming the initial human body parameterized model according to the body parameters of the target object to obtain a target human body parameterized model of a preset posture.

[0084] The body parameters of the target object have been extracted from the first video frame. When executing S3021, the body parameters of the target object are substituted into Formula 1 and Formula 2, and the SMPL model is deformed to obtain a target human body parameterized model of a preset posture corresponding to the target object.

[0085] S3022: Align the target human body parameterized model with the coordinate origin of the voxel space.

[0086] When executing S3022, the coordinate origin of the target human body parametric model is aligned with the coordinate origin of the voxel space. After alignment, the voxel space divides the entire target human body parametric model into front-to-back connected voxel blocks, and each voxel block has the same volume and the same length, width and height.

[0087] S3023: Generate a reference frame model with skin according to the SDF value of each vertex in the voxel space in the target human body parameterized model by using the set first deep neural network.

[0088] After aligning the target human body parameterized model with the coordinate origin of the voxel space, for each vertex in the target human body parameterized model, a signed distance field (SDF) value of the vertex in the voxel space can be determined.

[0089] Among them, the SDF value of the real surface of the target object in the voxel space is 0. If the SDF value is positive, it means it is outside the target object; if the SDF value is negative, it means it is inside the target object.

[0090] Assume that the first deep neural network is recorded as net1, and the network is composed of a fully connected multilayer perceptron (MLP). When executing S3023, the three-dimensional coordinates P (x, y, z) of each vertex in the voxel space and the preset network parameter η are input into the first deep neural network, and the isosurface with an SDF value of zero is learned through the first deep neural network to form an implicit expression of the reference frame model with skin, and then the reference frame model of the display expression is obtained through rendering.

[0091] The determination process of the reference frame model with skin is described in detail in Figure 6 , mainly includes the following steps:

[0092] S3023_1: Using a first deep neural network, determine the SDF value of each vertex in the voxel space according to the three-dimensional coordinates of each vertex in the target human body parameterized model in the voxel space.

[0093] After aligning the target human parametric model with the coordinate origin of the voxel space, the three-dimensional coordinates of each vertex of the target human parametric model in the corresponding voxel block in the voxel space can be determined, which is recorded as P(x, y, z). Each three-dimensional coordinate can reflect the spatial position relationship between the vertex and the voxel block, and based on the spatial position relationship, the SDF value of the vertex in the voxel space is obtained.

[0094] S3023_2: Extract the vertex set with SDF value 0 to obtain the implicit expression of the reference frame model with skin.

[0095] The set of vertices with SDF values ​​of zero constitutes a zero isosurface, which can be used as an implicit expression of the reference frame model with skin. The implicit expression formula is as follows:

[0096] S η = {P∈R 3 |f(P;η)=0} Formula 3

[0097] Among them, S η represents the set of vertices whose SDF value is 0, P is the three-dimensional coordinate of the vertex in the voxel space, η represents the network parameters of the first deep learning network, which can be optimized through learning, and f represents the first deep neural network.

[0098] S3023_3: Render the zero isosurface formed by the vertices in the vertex set to obtain a reference frame model with skin.

[0099] In the specific implementation, the marching cube algorithm is used to extract the three-dimensional mesh model of the target object by rendering the zero isosurface composed of a vertex set with zero SDF value, and obtain the reference frame model with skin for display expression.

[0100] S303: deforming the reference frame model according to the posture parameters of the target object contained in each video frame and the three-dimensional coordinates of each vertex in the reference frame model in the corresponding voxel block to obtain the target geometric model.

[0101] The following takes a video frame as an example. For the specific deformation process, see Figure 7 :

[0102] S3031: Mapping the reference frame model to the video frame according to the posture parameters of the target object contained in the video frame.

[0103] In an embodiment of the present application, a first deep neural network is used to generate a reference frame model with skin according to the initialization result as the initial reference frame model. When executing S3031, the implicit expression and explicit expression of the initial reference frame model are mapped to the video frame according to the body parameters of the target object contained in the video frame, and the three-dimensional coordinates of each vertex in the corresponding voxel block in the mapped reference frame model are determined. Among them, the three-dimensional coordinates can be input into the set second deep neural network as a conditional variable for deforming the mapped reference frame model.

[0104] S3032: Determine the non-rigid deformation relationship between the video frame and the reference frame model using a set second deep neural network according to the shape parameters of the target object contained in the video frame and the three-dimensional coordinates of each vertex in the mapped reference frame model in the corresponding voxel block.

[0105] Among them, the second deep neural network is similar to the first deep neural network and can also be composed of MLP full connection, denoted as net2.

[0106] When executing S3032, the shape parameters of the target object contained in the video frame and the three-dimensional coordinates of each vertex in the mapped reference frame model in the corresponding voxel block are respectively input into the second deep neural network, and the non-rigid deformation relationship d between the video frame and the reference frame model is output through the second deep neural network.

[0107] S3033: Deforming the reference frame model according to the non-rigid deformation relationship.

[0108] When executing S3033, the skin function W corresponding to the current posture parameters is used to determine the skin deformation field D = W(d(·)) corresponding to the video frame according to the non-rigid deformation relationship, and the implicit expression and explicit expression of the reference frame model are deformed according to the skin deformation field D to obtain the reference frame model corresponding to the current morphological parameters.

[0109] In the embodiment of the present application, in the process of deforming the reference frame model using each video frame, the loss of the reference frame model before and after deformation corresponding to the previous video frame is used to optimize the deformation parameters of the next video frame, that is, the non-rigid deformation relationship. For the specific process, see Figure 8 :

[0110] S3034: Projecting the deformed reference frame model into the video frame according to the parameters of the RGB camera.

[0111] When executing S3034, the reference frame model corresponding to the current posture parameter is projected into the video frame corresponding to the current posture parameter according to the intrinsic parameters and extrinsic parameters of the RGB camera.

[0112] S3035: Optimizing the implicit expression of the reference frame model according to the loss value between the outer contour of the reference frame model after projection and the outer contour of the target object segmented in the video frame.

[0113] When executing S3035, a portrait segmentation algorithm is used to extract the outer contour of the target object contained in the video frame, and the outer contour of the projected reference frame model is extracted, and the loss values ​​of the two are calculated. The network parameters of the first deep neural network are adjusted according to the loss values ​​to optimize the implicit expression of the reference frame model and improve the authenticity of the virtual character.

[0114] S3036: Determine each first intersection point of the non-rigid projection ray with the implicit expression of the optimized reference frame model, and each second intersection point of the non-rigid projection ray with the implicit expression of the optimized reference frame model.

[0115] When executing S3036, the network parameters of the first deep neural network are adjusted by the loss value, and after obtaining the implicit expression of the optimized reference frame model, each intersection point of the non-rigid projection ray with the implicit expression of the optimized reference frame model (recorded as the first intersection point) and each intersection point with the implicit expression of the reference frame model before optimization (recorded as the second intersection point) are determined. Among them, the SDF difference between the first intersection point and the second intersection point can fully reflect the detailed features of the reference frame model and improve the authenticity of the virtual character.

[0116] S3037: According to the posture parameters of the target object contained in the video frame, the voxel block corresponding to the SDF difference between the first intersection point and the second intersection point is transformed into the voxel space to optimize the non-rigid deformation relationship of the next video frame to deform the reference frame model.

[0117] When executing S3037, the voxel block corresponding to the SDF difference between the first intersection and the second intersection is transformed into the voxel space through the posture parameters of the target object contained in the video frame to adjust the network parameters of the second deep neural network. In this way, when the non-rigid deformation relationship of the implicit expression and the explicit expression of the reference frame model of the next video is determined according to the second deep neural network, the accuracy of the non-rigid deformation relationship can be improved, thereby improving the accuracy of the virtual character.

[0118] repeat Figure 7 and Figure 8 The method steps are as follows, until the deformation of the reference frame model by the last video frame is completed, and the final target geometric model is obtained. Fig. 9 , the target geometric model has topological structure consistency.

[0119] In an embodiment of the present application, when the reference frame model is deformed according to the posture parameters of the target object contained in each video frame, the network parameters of the first deep neural network and the second deep neural network are adjusted respectively. Each adjustment is beneficial to improving the accuracy of the non-rigid deformation relationship when the reference frame model is deformed in the next frame, so that the deformed reference frame model is consistent with the posture of the target object, thereby improving the authenticity of the virtual character.

[0120] S304: Perform texture mapping on the target geometric model according to the RGB information of each video frame to obtain a virtual character of the target object.

[0121] In an embodiment of the present application, a video of the target object rotating in a preset posture is collected, which contains all RGB information of the target object at multiple angles. When executing S304, a complete texture map of the target object can be obtained based on the RGB information contained in each video frame. After texture mapping of the target geometric model, a virtual character of the target object is obtained. The virtual character has the same skin as the target object and can truly reflect the appearance of the target object.

[0122] In S304, when the target geometric model is textured, it is necessary to determine the correspondence between the vertices of the target geometric model and the pixels of the texture image. Fig.10 , the specific process is as follows:

[0123] S3041: Generate in advance a first blank texture map with a fixed resolution.

[0124] For example, the resolution of the first texture map is set to 2048*2048px. Since the first texture map is blank, it can be subsequently rendered using the RGB information in each video frame to obtain a complete texture map consistent with the real skin of the target object.

[0125] S3042: Rasterize the target geometric model, and determine a first correspondence between the vertices contained in each triangular face after rasterization and the pixel points in a prefabricated second texture map, wherein the resolution size of the second texture map is the same as that of the first texture map.

[0126] In S3042, by rasterizing the target geometric model, a plurality of triangular patches can be obtained, each triangular patch includes three vertices, each vertex corresponds to an index, and the index can be the corresponding triangular patch and the centroid coordinates of the triangular patch. The vertices included in each triangular patch are upsampled to determine a first correspondence between each vertex and a pixel point in the second texture map, wherein the first correspondence can be represented by the index of the vertex and the coordinates of the pixel point.

[0127] S3043: Migrate the first correspondence relationship to the first texture map to obtain a second correspondence relationship between the vertices of the target geometric model and the pixels of the first texture map.

[0128] In S3043, since the resolution sizes of the first texture map and the second texture map are the same, that is, the vertices of the target geometric model have corresponding pixels in the first texture map and also have corresponding pixels in the second texture map. Therefore, the first correspondence between the vertices of the target geometric model and the pixels of the second texture map can be migrated to the first texture map, and the second correspondence between the vertices of the target geometric model and the pixels of the first texture map can be obtained and stored.

[0129] Furthermore, after rendering the first texture map according to the RGB information extracted from each video frame, the complete texture data of the target object can be obtained, and then the texture mapping of the target geometric model is completed by using the second correspondence between the vertices of the target geometric model and the pixels of the first texture map. Fig.11 , mainly includes the following steps:

[0130] S3044: For each video frame, determine a third correspondence between pixel points in the video frame and vertices of the target geometric model.

[0131] When executing S3044, the skeleton corresponding to the target geometric model is extracted, and the target geometric model is mapped to the video frame through the intrinsic parameters and extrinsic parameters of the RGB camera, and the third corresponding relationship between the pixel points in the video frame and the vertices of the target geometric model is determined.

[0132] S3045: Rendering a blank first texture map using RGB information of pixels in the video frame according to the third corresponding relationship and the second corresponding relationship.

[0133] In S3045, based on the third correspondence between the pixel points in the video frame and the vertices of the target geometric model, and the second correspondence between the vertices of the target geometric model and the pixel points of the first texture map, the pixel points in the video frame corresponding to the pixel points in the first texture map can be determined, and the RGB information of the pixel points in the video frame can be assigned to the corresponding pixel points in the first texture map to complete the rendering of the first texture map.

[0134] S3046: Perform texture mapping on the target geometric model according to the first texture map rendered after each video frame, to obtain a virtual character of the target object.

[0135] In S3046, after all RGB information of the pixels in each video frame is assigned to the corresponding pixels in the first texture map, the first texture map contains the complete texture data of the target object, so that after texture mapping of the target geometric model according to the first texture map, the virtual character of the target object can be obtained, see Fig.12 ,Since the first texture map contains the real skin (clothes, hair, etc.) data of the target object, the appearance of the virtual character is consistent with the target object, the authenticity of the target object is improved.

[0136] In the method for generating a virtual character provided in the embodiment of the present application, a civilian-grade RGB camera is used to collect and store a video of a target object rotating in a preset posture, a front image with rich detail information is used as the first video frame to extract the body parameters of the target object, and the initial human body parameterized model with consistent topological structure is voxelized with the body parameters to generate a reference frame model with skin; after obtaining the reference frame model with skin, the reference frame model is deformed according to the posture parameters contained in each video frame and the three-dimensional coordinates of each vertex in the reference frame model in the corresponding voxel block to obtain a target geometric model, and during the deformation process, the loss of the reference frame model before and after deformation corresponding to the previous video frame is used to optimize the deformation parameters of the next video frame, so that the target geometric model is more consistent with the target object, and the authenticity of the target geometric model is improved, so that after the target geometric model is texture mapped according to the RGB information of each video frame, a more realistic virtual character is obtained, and the skin of the virtual character can truly depict the appearance of the target object. During the entire generation process, the optimization of the deformation parameters is automatically performed without the participation of animators, saving manpower and material resources, and improving the generation efficiency of virtual characters.

[0137] After obtaining the virtual character of the target object, you can use the virtual character for remote 3D communication to enhance the face-to-face immersion during the interaction. Fig.13 , mainly includes the following steps:

[0138] S305: Acquire an RGB image of the target object captured by the RGB camera during the interaction process.

[0139] During the interaction, the target object can move in any posture, and the RGB image of the interaction process is collected by the RGB camera in real time and transmitted to the GPU of the interactive device.

[0140] S306: Extracting motion data of the target object according to the RGB image.

[0141] Through the motion capture algorithm, the target object skeleton nodes are extracted from the RGB image, and the motion data of the skeleton nodes, such as the rotation angle of the skeleton nodes, the coordinates of the skeleton nodes, etc., are obtained.

[0142] S307: According to the motion data, the virtual character is driven to move to match the real motion of the target object.

[0143] The obtained motion data of the skeleton nodes is used to drive the virtual character to move according to the driving relationship between the skeleton nodes and the model vertices, so that the action of the virtual character after the movement is consistent with the current real action of the target.

[0144] In the embodiments of the present application, the motion data of the target object occupies a smaller amount of data than the three-dimensional model data. Under the conditions of the existing network bandwidth, it meets the real-time requirements of remote three-dimensional communication, reduces the lag in the communication process, and improves the user's immersive experience.

[0145] Based on the same technical concept, an embodiment of the present application provides an interactive device, which can execute the steps of the virtual character generation method provided in the above embodiment and achieve the same technical effect, which will not be repeated here.

[0146] See also Fig.14A , the interactive device includes a processor 1401, a memory 1402, a display 1403 and a communication interface 1404, wherein the communication interface 1404, the display 1403, the memory 1402 and the processor 1401 are connected via a bus 1405;

[0147] The memory 1402 includes a data storage unit and a program storage unit. The program storage unit stores computer program instructions. The processor 1401 executes the following operations according to the computer program:

[0148] Through the communication interface 1404, a video of the target object rotating in a preset posture captured by the RGB camera is obtained and stored in the data storage unit, and the front image of the target object is used as the first video frame to extract the body parameters of the target object;

[0149] Initializing a voxel space of a set resolution according to an initial human body parameterized model and the body parameters of the target object, and generating a reference frame model with skin according to the initialization result;

[0150] According to the posture parameters of the target object contained in each video frame and the three-dimensional coordinates of each vertex in the reference frame model in the corresponding voxel block, the reference frame model is deformed to obtain the target geometric model; during the deformation process, the loss of the reference frame model corresponding to the previous video frame before and after deformation is used to optimize the deformation parameters of the next video frame;

[0151] The target geometric model is texture mapped according to the RGB information of each video frame to obtain a virtual character of the target object, which is then displayed by the display 1403 .

[0152] Optionally, the processor 1401 initializes a voxel space of a set resolution according to the initial human body parameterized model and the body parameters, and generates a reference frame model with skin according to the initialization result, and the specific operation is:

[0153] According to the body parameters, the initial human body parameterized model is deformed to obtain a target human body parameterized model of a preset posture;

[0154] Aligning the target human body parameterized model with the coordinate origin of the voxel space;

[0155] A reference frame model with skin is generated by using a set first deep neural network according to the SDF value of each vertex in the target human body parameterized model in the voxel space.

[0156] Optionally, the processor 1401 generates a reference frame model with skin according to the SDF value of each vertex in the target human body parameterized model in the voxel space by using a set first deep neural network, and the specific operation is:

[0157] Determine the SDF value of each vertex in the voxel space according to the three-dimensional coordinates of each vertex in the target human body parameterized model in the voxel space;

[0158] Extract the vertex set with SDF value 0 to obtain the implicit expression of the reference frame model with skin;

[0159] The zero isosurface formed by the vertices in the vertex set is rendered to generate a reference frame model with skin for display expression.

[0160] Optionally, the processor 1401 deforms the reference frame model according to the posture parameters of the target object contained in each video frame and the three-dimensional coordinates of each vertex in the reference frame model in the corresponding voxel block, and the specific operation is:

[0161] For each video frame, mapping the reference frame model to the video frame according to the posture parameters of the target object contained in the video frame;

[0162] Determine, using a set second deep neural network, a non-rigid deformation relationship between the video frame and the reference frame model according to the posture parameters and the three-dimensional coordinates of each vertex in the mapped reference frame model in the corresponding voxel block;

[0163] The reference frame model is deformed according to the non-rigid deformation relationship.

[0164] Optionally, after deforming the reference frame model for each video frame, the processor 1401 further executes:

[0165] Projecting the deformed reference frame model into the video frame according to the parameters of the RGB camera;

[0166] Optimizing the implicit expression of the reference frame model according to a loss value between an outer contour of the reference frame model after projection and an outer contour of the target object obtained by segmentation in the video frame;

[0167] Determine each first intersection point of the non-rigid projection ray with the implicit expression of the reference frame model after optimization, and each second intersection point of the non-rigid projection ray with the implicit expression of the reference frame model before optimization;

[0168] According to the posture parameters of the target object contained in the video frame, the voxel block corresponding to the SDF difference between the first intersection point and the second intersection point is transformed into the voxel space to optimize the non-rigid deformation relationship of the next video frame to deform the reference frame model.

[0169] Optionally, the processor 1401 performs texture mapping on the target geometric model according to the RGB information of each video frame to obtain a virtual character of the target object, and the specific operation is:

[0170] Rasterizing the target geometric model, determining a first correspondence between vertices included in each triangular facet after rasterization and pixel points in a prefabricated second texture map, wherein the resolution of the second texture map is the same as that of the first texture map, and the first texture map is a prefabricated blank texture map;

[0171] Migrating the first correspondence relationship to the first texture map to obtain a second correspondence relationship between vertices of the target geometric model and pixels of the first texture map;

[0172] For each video frame, determining a third correspondence between pixels in the video frame and vertices of the target geometric model;

[0173] Rendering the first texture map using RGB information of pixels in the video frame according to the third corresponding relationship and the second corresponding relationship;

[0174] The target geometric model is texture mapped according to the first texture map rendered after each video frame is rendered to obtain a virtual character of the target object.

[0175] Optionally, after obtaining the virtual character of the target object, the processor 1401 further executes:

[0176] Acquire an RGB image of the target object captured by the RGB camera during the interaction process;

[0177] Extracting motion data of the target object according to the RGB image;

[0178] According to the motion data, the virtual character is driven to move to match the real motion of the target object.

[0179] Optionally, the processor 1401 determines the morphological parameters and the posture parameters in the following manner:

[0180] For the first video frame, extracting a target skeleton and a first human body contour of the target object;

[0181] Initializing the shape parameters and posture parameters of the target skeleton, and deforming the initial human body parameterized model through a skinning function;

[0182] Extracting a reference skeleton of the deformed human body parameterized model, and determining a skeleton loss value between the reference skeleton and the target skeleton;

[0183] According to the intrinsic parameters of the RGB camera, the reference skeleton is projected into the first video frame to optimize the skeleton loss value, and the extrinsic parameters of the RGB camera and new posture parameters are obtained;

[0184] According to the new posture parameters and the external parameters, the vertices of the deformed human body parameterized model are projected into the first video frame to obtain a second human body contour;

[0185] updating body parameters and posture parameters according to contour loss values ​​of the first human body contour and the second human body contour;

[0186] The external parameters of the RGB camera and the body parameters of the target object are fixed, and for non-first video frames, the skeleton loss value and the contour loss value are re-determined to determine the posture parameters corresponding to the non-first video frames.

[0187] It should be noted that Fig.14AIt is only the necessary hardware required for the detection device to implement the method for generating a virtual character provided in the embodiment of the present application. Optionally, the interactive device also includes conventional hardware such as a speaker and an audio processor.

[0188] In another optional embodiment, the interactive device itself may integrate an RGB camera 1406, see Fig. 14B , other hardware and Fig.14A The same, no need to repeat here.

[0189] The embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete the method of the aforementioned embodiment.

[0190] The embodiments of the present application also provide a computer program product for storing a computer program, wherein the computer program is used to execute the method of the aforementioned embodiment.

[0191] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0192] The present application is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0193] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0194] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0195] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for generating a virtual character, characterized in that: include: Obtain a video captured by an RGB camera of a target object rotating in a preset posture, and use the front image of the target object as the first video frame to extract the shape parameters of the target object; Initializing a voxel space of a set resolution according to the initial human body parameterized model and the body parameters, and generating a reference frame model with skin according to the initialization result; According to the posture parameters of the target object contained in each video frame and the three-dimensional coordinates of each vertex in the reference frame model in the corresponding voxel block, the reference frame model is deformed to obtain the target geometric model; during the deformation process, the loss of the reference frame model corresponding to the previous video frame before and after deformation is used to optimize the deformation parameters of the next video frame; The target geometric model is texture mapped according to the RGB information of each video frame to obtain a virtual character of the target object; wherein the body parameters and posture parameters are determined in the following manner: For the first video frame, extracting a target skeleton and a first human body contour of the target object; Initializing the body parameters and posture parameters of the target skeleton, deforming the initial human body parameterized model through a skinning function; extracting a reference skeleton of the deformed human body parameterized model, and determining a skeleton loss value between the reference skeleton and the target skeleton; According to the intrinsic parameters of the RGB camera, the reference skeleton is projected into the first video frame to optimize the skeleton loss value, and the extrinsic parameters of the RGB camera and new posture parameters are obtained; According to the new posture parameters and the external parameters, the vertices of the deformed human body parameterized model are projected into the first video frame to obtain a second human body contour; According to the contour loss values ​​of the first human contour and the second human contour, the body parameters and the posture parameters are updated; the external parameters of the RGB camera and the body parameters of the target object are fixed, and for non-first video frames, the skeleton loss value and the contour loss value are re-determined to determine the posture parameters corresponding to the non-first video frames.

2. The method according to claim 1, characterized in that The step of initializing a voxel space of a set resolution according to the initial human body parameterized model and the body parameters, and generating a reference frame model with skin according to the initialization result, comprises: According to the body parameters, the initial human body parameterized model is deformed to obtain a target human body parameterized model of a preset posture; Aligning the target human body parameterized model with the coordinate origin of the voxel space; A reference frame model with skin is generated by using a set first deep neural network according to the SDF value of each vertex in the target human body parameterized model in the voxel space.

3. The method according to claim 2, characterized in that The method of using the set first deep neural network to generate a reference frame model with skin according to the SDF value of each vertex in the voxel space in the target human body parameterized model includes: Determine the SDF value of each vertex in the voxel space according to the three-dimensional coordinates of each vertex in the target human body parameterized model in the voxel space; Extract the vertex set with SDF value O to obtain the implicit expression of the reference frame model with skin; The zero isosurface formed by the vertices in the vertex set is rendered to generate a reference frame model with skin for display expression.

4. The method according to claim 1, characterized in that The step of deforming the reference frame model according to the posture parameters of the target object contained in each video frame and the three-dimensional coordinates of each vertex in the reference frame model in the corresponding voxel block comprises: For each video frame, mapping the reference frame model to the video frame according to the posture parameters of the target object contained in the video frame; Determine, using a set second deep neural network, a non-rigid deformation relationship between the video frame and the reference frame model according to the posture parameters and the three-dimensional coordinates of each vertex in the mapped reference frame model in the corresponding voxel block; The reference frame model is deformed according to the non-rigid deformation relationship.

5. The method according to claim 1, characterized in that After deforming the reference frame model for each video frame, the method further includes: projecting the deformed reference frame model into the video frame according to the parameters of the RGB camera; Optimizing the implicit expression of the reference frame model according to a loss value between an outer contour of the reference frame model after projection and an outer contour of the target object obtained by segmentation in the video frame; Determine each first intersection point of the non-rigid projection ray with the implicit expression of the reference frame model after optimization, and each second intersection point of the non-rigid projection ray with the implicit expression of the reference frame model before optimization; According to the posture parameters of the target object contained in the video frame, the voxel block corresponding to the SDF difference between the first intersection point and the second intersection point is transformed into the voxel space to optimize the non-rigid deformation relationship of the next video frame to deform the reference frame model.

6. The method according to claim 1, characterized in that The step of performing texture mapping on the target geometric model according to the RGB information of each video frame to obtain the virtual character of the target object includes: The target geometric model is rasterized, and a first correspondence between vertices contained in each triangle patch after rasterization and pixel points in a prefabricated second texture map is determined, wherein the resolution of the second texture map is the same as the resolution of the first texture map, and the first texture map is a prefabricated blank texture map. Migrating the first correspondence relationship to the first texture map to obtain a second correspondence relationship between vertices of the target geometric model and pixels of the first texture map; For each video frame, determining a third correspondence between pixels in the video frame and vertices of the target geometric model; According to the third corresponding relationship and the second corresponding relationship, the first texture image 2 is rendered using the RGB information of the pixel points in the video frame. The target geometric model is texture mapped according to the first texture map rendered after each video frame is rendered to obtain a virtual character of the target object.

7. The method according to any one of claims 1 to 6, characterized in that After obtaining the virtual character of the target object, the method further includes: Acquire an RGB image of the target object captured by the RGB camera during the interaction process; Extracting motion data of the target object according to the RGB image; According to the motion data, the virtual character is driven to move to match the real motion of the target object.

8. An interactive device, characterized in that: Used to generate a virtual character for remote interaction, the interactive device includes a processor, a memory, a display and a communication interface, and the communication interface, the display, the memory and the processor are connected via a bus; The memory includes a data storage unit and a program storage unit, the program storage unit stores computer program instructions, and the processor executes the following operations according to the computer program: Through the communication interface, a video of the target object rotating in a preset posture captured by the RGB camera is obtained and stored in the data storage unit, and the front image of the target object is used as the first video frame to extract the body parameters of the target object; Initializing a voxel space of a set resolution according to an initial human body parameterized model and the body parameters of the target object, and generating a reference frame model with skin according to the initialization result; According to the posture parameters of the target object contained in each video frame and the three-dimensional coordinates of each vertex in the reference frame model in the corresponding voxel block, the reference frame model is deformed to obtain the target geometric model; during the deformation process, the loss of the reference frame model corresponding to the previous video frame before and after deformation is used to optimize the deformation parameters of the next video frame; The target geometric model is texture mapped according to the RGB information of each video frame to obtain a virtual character of the target object, and the virtual character is displayed by the display; wherein the body parameters and posture parameters are determined by the following method: For the first video frame, extracting a target skeleton and a first human body contour of the target object; Initializing the body parameters and posture parameters of the target skeleton, deforming the initial human body parameterized model through a skinning function; extracting a reference skeleton of the deformed human body parameterized model, and determining a skeleton loss value between the reference skeleton and the target skeleton; According to the intrinsic parameters of the RGB camera, the reference skeleton is projected into the first video frame to optimize the skeleton loss value, and the extrinsic parameters of the RGB camera and new posture parameters are obtained; According to the new posture parameters and the external parameters, the vertices of the deformed human body parameterized model are projected into the first video frame to obtain a second human body contour; updating body parameters and posture parameters according to contour loss values ​​of the first human body contour and the second human body contour; The external parameters of the RGB camera and the body parameters of the target object are fixed, and for non-first video frames, the skeleton loss value and the contour loss value are re-determined to determine the posture parameters corresponding to the non-first video frames.

9. The interactive device according to claim 8, characterized in that: After obtaining the virtual character of the target object, the processor further executes: Acquire, through the communication interface, an RGB image of the target object captured by the RGB camera during the interaction process; Extracting motion data of the target object according to the RGB image; According to the motion data, the virtual character is driven to move to match the real posture of the target object, and the driven virtual character is displayed on the display.

Images