A three-dimensional model reconstruction method, device and equipment

Abstract

The present application provides a three-dimensional model reconstruction method, device and equipment, which relates to the field of VR / AR technology. The method includes: obtaining a captured two-dimensional image, inputting the two-dimensional image into a feature extraction module, wherein the sample image is input into a feature extraction layer to extract features of the sample image, and a reconstruction layer is used to reconstruct and output a three-dimensional model according to the extracted image feature parameters, and the parameters of the feature extraction layer and the reconstruction layer are adjusted with the output of the sample three-dimensional model as the goal. When the adjustment is completed, the feature extraction layer is used as a feature extraction module; the image feature parameters of the two-dimensional image are extracted using the feature extraction module, and the image feature parameters include shape parameters, expression parameters and posture parameters; and the image feature parameters are transmitted to the cloud. The present application can greatly reduce the amount of data, effectively solve the problem of large-scale calculations at the acquisition end and the problem of high data pressure at the transmission end, and ensure good rendering quality.

Description

Technical Field

[0001] The present invention relates to the field of AR / VR technology, and in particular to a three-dimensional model reconstruction method, device and equipment. Background Art

[0002] The remote 3D communication and interaction system can generally be divided into three ends: the acquisition end, the cloud end, and the display end. The acquisition end is responsible for remote data acquisition, the cloud end is responsible for reconstructing the 3D model data, and the display end obtains data from the cloud end for rendering.

[0003] At present, the demand for high-precision models is becoming more and more widespread. High-precision 3D reconstruction of human body models mainly includes the following:

[0004] Collecting data for three-dimensional reconstruction through optical scanners requires the subject to remain still for several seconds or even minutes during the entire scanning process. By splicing high-precision three-dimensional scanning information from multiple angles, a high-precision static three-dimensional model of the human body can be reconstructed.

[0005] The 3D face reconstruction algorithm based on statistical model builds a face database by sampling and statistically analyzing a large number of samples. The main idea is to assume that any 3D face can be represented by a linear transformation on an average 3D face model, iteratively match and compare the average model with the input image, and deform the average model to match the input image, thereby achieving 3D reconstruction.

[0006] The above-mentioned high-precision 3D reconstruction method of the human body model is to complete the 3D reconstruction in the cloud and transmit the 3D data of the reconstructed high-precision human body model to the display end. The refinement of real-time 3D reconstruction will cause the amount of 3D geometric data to increase, which will cause transmission pressure in the cloud and cause visual delay in the rendering application end. Therefore, it is necessary to provide a solution that effectively guarantees the accuracy of the reconstructed model, reduces the transmission bandwidth occupancy, and ensures low latency. Summary of the invention

[0007] The embodiments of the present application provide a three-dimensional model reconstruction method, device and equipment, which extract features from a captured two-dimensional image to obtain image feature parameters of the two-dimensional image, and transmit the image feature parameters to the cloud, thereby realizing small-scale data transmission.

[0008] In a first aspect, an embodiment of the present application provides a three-dimensional model reconstruction method, which is applied to a collection end, and the method includes:

[0009] Acquire the collected two-dimensional image, and input the two-dimensional image into the feature extraction module, wherein the sample image is input into the feature extraction layer to extract features of the sample image, and the reconstruction layer is used to reconstruct and output the three-dimensional model according to the extracted image feature parameters, and the parameters of the feature extraction layer and the reconstruction layer are adjusted with the output of the sample three-dimensional model as the goal, and the feature extraction layer is used as the feature extraction module after the adjustment is completed;

[0010] Extracting image feature parameters of the two-dimensional image using the feature extraction module, wherein the image feature parameters include shape parameters, expression parameters and posture parameters;

[0011] The image feature parameters are transmitted to the cloud.

[0012] In a possible implementation, extracting image feature parameters of the two-dimensional image by using the feature extraction module includes:

[0013] The shape parameters, expression parameters and posture parameters of the two-dimensional image are extracted using the feature extraction module:

[0014] Transmitting the image feature parameters to the cloud includes:

[0015] When determining that the two-dimensional image is a key frame image, transmitting the shape parameters, expression parameters and posture parameters to the cloud;

[0016] When it is determined that the two-dimensional image is a non-key frame image, the expression parameters and the posture parameters are transmitted to the cloud.

[0017] In a possible implementation manner, transmitting the image feature parameters to the cloud includes:

[0018] Using a high-frequency noise filtering algorithm to filter the same image feature parameter of at least one two-dimensional image, the high-frequency noise filtering algorithm includes sliding average, exponential sliding average method and SG filtering method;

[0019] The filtered image feature parameters are transmitted to the cloud.

[0020] In a possible implementation, the parameters of the feature extraction layer and the reconstruction layer are adjusted in the following manner:

[0021] Input the sample image into the feature extraction layer to extract features of the sample image, and obtain an image feature vector corresponding to the image feature parameters;

[0022] Input the image feature vector to the reconstruction layer, use the basis function of the reconstruction layer to determine the weight parameters corresponding to different vertices in the reference 3D model based on the image feature vector, and calculate the coordinates of each vertex in the predicted 3D model according to the weight parameters, and determine and output the predicted 3D model based on the calculated coordinates of each vertex and the vertices of the triangle patch index;

[0023] The loss function value is determined according to the difference between the predicted three-dimensional model and the sample three-dimensional model, and the parameters of the feature extraction layer and the reconstruction layer are adjusted according to the loss function value.

[0024] In a second aspect, an embodiment of the present application provides a three-dimensional model reconstruction method, which is applied to the cloud, and the method includes:

[0025] Receiving image feature parameters of the two-dimensional image transmitted by the acquisition end, the image feature parameters including shape parameters, expression parameters and posture parameters;

[0026] The image feature parameters are encoded and decoded and then transmitted to the rendering end.

[0027] In a possible implementation manner, encoding and decoding the image feature parameters and then transmitting them to the rendering end includes:

[0028] Using a high-frequency noise filtering algorithm to filter the same image feature parameter of at least one two-dimensional image, the high-frequency noise filtering algorithm includes sliding average, exponential sliding average method and SG filtering method;

[0029] The filtered image feature parameters are encoded and decoded and transmitted to the rendering end.

[0030] In a third aspect, an embodiment of the present application provides a three-dimensional model reconstruction method, which is applied to a rendering end, and the method includes:

[0031] Receiving image feature parameters of a two-dimensional image transmitted from the cloud, wherein the image feature parameters include shape parameters, expression parameters, and posture parameters;

[0032] The image feature parameters are input into a three-dimensional reconstruction module, and a reconstructed three-dimensional model is obtained by using the three-dimensional reconstruction module, wherein features of the sample image are extracted by inputting the sample image into a feature extraction layer, and a three-dimensional model is reconstructed and outputted according to the extracted image feature parameters by using a reconstruction layer, and parameters of the feature extraction layer and the reconstruction layer are adjusted with the goal of outputting the sample three-dimensional model, and at the end of the adjustment, the reconstruction layer is used as a three-dimensional reconstruction module.

[0033] In a fourth aspect, an embodiment of the present application provides a three-dimensional model reconstruction device, which is applied to a collection end, and the device includes:

[0034] An image acquisition module is used to acquire a captured two-dimensional image and input the two-dimensional image into a feature extraction module, wherein the sample image is input into a feature extraction layer to extract features of the sample image, a reconstruction layer is used to reconstruct and output a three-dimensional model according to the extracted image feature parameters, and the parameters of the feature extraction layer and the reconstruction layer are adjusted with the output of the sample three-dimensional model as the goal, and the feature extraction layer is used as a feature extraction module when the adjustment is completed;

[0035] A feature extraction module, used to extract image feature parameters of the two-dimensional image using the feature extraction module, wherein the image feature parameters include shape parameters, expression parameters and posture parameters;

[0036] The parameter transmission module is used to transmit the image feature parameters to the cloud.

[0037] In a fifth aspect, an embodiment of the present application provides a three-dimensional model reconstruction device, which is applied in the cloud, and the device includes:

[0038] A parameter receiving module, used for receiving image feature parameters of a two-dimensional image transmitted by a collection end, wherein the image feature parameters include shape parameters, expression parameters and posture parameters;

[0039] The parameter transmission module is used to transmit the image feature parameters to the rendering end after encoding and decoding.

[0040] In a sixth aspect, an embodiment of the present application provides a three-dimensional model reconstruction device, which is applied to a rendering end, and the device includes:

[0041] A parameter receiving module, used to receive image feature parameters of a two-dimensional image transmitted from the cloud, wherein the image feature parameters include shape parameters, expression parameters and posture parameters;

[0042] A three-dimensional reconstruction module is used to input the image feature parameters into the three-dimensional reconstruction module, and use the three-dimensional reconstruction module to obtain a reconstructed three-dimensional model, wherein the sample image is input into a feature extraction layer to extract features of the sample image, and the reconstruction layer is used to reconstruct and output the three-dimensional model according to the extracted image feature parameters, and the parameters of the feature extraction layer and the reconstruction layer are adjusted with the goal of outputting the sample three-dimensional model. When the adjustment is completed, the reconstruction layer is used as the three-dimensional reconstruction module.

[0043] In a seventh aspect, an embodiment of the present application provides a three-dimensional model reconstruction device, the device comprising:

[0044] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform any one of the three-dimensional model reconstruction methods.

[0045] In an eighth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program product is used to enable a computer to execute any one of the three-dimensional model reconstruction methods.

[0046] The embodiments of the present application provide a three-dimensional model reconstruction method, device and equipment, which extract features from a captured two-dimensional image to obtain image feature parameters of the two-dimensional image, and transmit the image feature parameters to the cloud, thereby effectively solving the problems of large-scale calculations on the acquisition end, high data pressure on the transmission end, and delay and realism problems on the rendering end. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a remote communication system according to an exemplary embodiment of the present invention;

[0048] Figure 2 A schematic diagram of an application scenario of a three-dimensional model reconstruction method according to an exemplary embodiment of the present invention;

[0049] Figure 3 A traditional rendering method according to an exemplary embodiment of the present invention is exemplified;

[0050] Figure 4 A schematic diagram of a flow chart of a three-dimensional model reconstruction method according to an exemplary embodiment of the present invention;

[0051] Figure 5 A schematic diagram of a flow chart of a three-dimensional model reconstruction method according to an exemplary embodiment of the present invention;

[0052] Figure 6 A schematic diagram of a flow chart of a three-dimensional model reconstruction method according to an exemplary embodiment of the present invention;

[0053] Figure 7 A schematic diagram of a parameter transfer and optimization process according to an exemplary embodiment of the present invention;

[0054] Figure 8 A three-dimensional model reconstruction device according to an exemplary embodiment of the present invention is intended;

[0055] Fig. 9 A three-dimensional model reconstruction device according to an exemplary embodiment of the present invention is intended;

[0056] Fig.10 A three-dimensional model reconstruction device according to an exemplary embodiment of the present invention is intended;

[0057] Fig.11The present invention is intended to provide a three-dimensional model reconstruction device according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The three-dimensional model reconstruction method provided in the embodiment of the present application is applied to remote communication scenarios, such as Figure 1 The figure shows a schematic diagram of a remote communication system, which includes a collection terminal 101, a cloud 102, and a rendering terminal 103. The collection terminal includes an RGBD camera 101_1 and a host / workstation 101_2. The RGBD camera 101_1 is used to collect two-dimensional images; the host / workstation 101_2 is used to process the collected two-dimensional images; the cloud 102 is used for data transmission; and the rendering terminal 103 includes a mobile phone, a TV, and a VR / AR helmet, which is used to reconstruct a three-dimensional model.

[0060] It should be noted that Figure 1 The system architecture shown can be deployed according to different usage scenarios. For example, in a live broadcast scenario, the host sets up the acquisition device of the system, and the user can browse the reconstructed model through VR / AR glasses, mobile phones, TVs, etc.; for another example, in a conference scenario, the two conference rooms of a remote meeting need to be equipped with acquisition devices and rendering display devices at the same time to carry out real-time three-dimensional communication in the two conference rooms.

[0061] based on Figure 1 The system architecture shown, Figure 2 The following is a schematic diagram of an application scenario provided by an embodiment of the present application. Figure 2 As shown, user terminals 1 to 4 perform real-time remote three-dimensional communication. User terminals 1 to 4 are respectively equipped with acquisition terminal devices (including RGBD cameras and master stations / workstations) and rendering terminal devices (including all or part of TVs, mobile phones, and VR / AR head displays). During the remote three-dimensional communication process, the three-dimensional reconstructed model of user terminal 1 can be uploaded to the cloud server, and user terminals 2 to 4 download the three-dimensional reconstructed model of user terminal 1 from the cloud server and display it synchronously. Similarly, user terminals 1, 3, and 4 can also synchronously display the three-dimensional reconstructed model of user terminal 2, and so on.

[0062] It should be noted that Figure 2 This is only an example of multi-person remote three-dimensional communication. The embodiment of the present application does not impose any restrictive requirements on the number of user terminals of the remote three-dimensional communication.

[0063] like Figure 3 The figure shows the traditional rendering method, which uses the acquisition end to collect RGB / RGBD data, reconstructs the three-dimensional model at the acquisition end, transmits the reconstructed high-precision three-dimensional data of the human body model to the cloud, and transmits it to the rendering end after encoding and decoding by the cloud server.

[0064] In order to solve the problems of heavy cloud transmission pressure and visual delay on the rendering end caused by the above transmission methods, the following Figure 4 The method shown:

[0065] S401: Acquire a collected two-dimensional image, and input the two-dimensional image into a feature extraction module, wherein features of the sample image are extracted by inputting the sample image into a feature extraction layer, and a three-dimensional model is reconstructed and outputted according to the extracted image feature parameters using a reconstruction layer, and parameters of the feature extraction layer and the reconstruction layer are adjusted with the goal of outputting the sample three-dimensional model. When the adjustment is completed, the feature extraction layer is used as a feature extraction module.

[0066] In a possible implementation, the embodiment of the present application is used to reconstruct a three-dimensional model of the head. An RGB / RGBD camera is used, including but not limited to Azure Kinect DK or Realsense camera. An RGB two-dimensional image of the head and a depth two-dimensional image are collected, and the depth two-dimensional image describes the distance between each part of the head and the camera. When the depth two-dimensional image is not collected, the three-dimensional model reconstruction method provided in this embodiment can also be completed using only the RGB two-dimensional image.

[0067] The collected two-dimensional image is input into the feature extraction module to extract image parameters, including posture parameters and expression parameters (represented in the form of multidimensional arrays). Posture parameters include head turning left, right, lowering the head, raising the head, etc.; expression parameters include: smiling, frowning, opening the mouth, etc.

[0068] The feature extraction module is deployed at the acquisition end, and the 3D reconstruction module is deployed at the rendering end. During training, a batch of samples can be used in the same device to implement the training of the feature extraction module and the 3D reconstruction module, and the samples can be input into a network model including a feature extraction layer and a reconstruction layer. The parameters of the feature extraction layer and the reconstruction layer are adjusted with the goal of outputting a sample 3D model. At the end of the adjustment, the feature extraction layer is used as the feature extraction module.

[0069] In the process of collecting two-dimensional images, two-dimensional images of the head in multiple directions are collected. For example, when nodding, a front two-dimensional image and a side two-dimensional image of the head are collected.

[0070] S402: Utilizing the feature extraction module to extract image feature parameters of the two-dimensional image, wherein the image feature parameters include shape parameters, expression parameters, and posture parameters.

[0071] In a possible implementation, extracting image feature parameters of the two-dimensional image by using the feature extraction module includes:

[0072] The feature extraction module is used to extract shape parameters, expression parameters and posture parameters of the two-dimensional image.

[0073] In the process of extracting image feature parameters, shape parameters, expression parameters and posture parameters are extracted from the two-dimensional image. Shape parameters are used to describe the geometric features of the head. When a person's head moves or makes facial expressions, its geometric features will not change significantly. Therefore, although shape parameters are extracted for each frame of the two-dimensional image, in the subsequent image feature parameter transmission or three-dimensional model reconstruction process, not every frame of the two-dimensional image shape parameters will be used.

[0074] S403: Transmit the image feature parameters to the cloud.

[0075] In a possible implementation, transmitting the image feature parameters to the cloud includes:

[0076] When determining that the two-dimensional image is a key frame image, transmitting the shape parameters, expression parameters and posture parameters to the cloud;

[0077] When it is determined that the two-dimensional image is a non-key frame image, the expression parameters and the posture parameters are transmitted to the cloud.

[0078] There will be differences in the shape parameters extracted from each frame of the two-dimensional image, but the shape of the human head will not change due to changes in posture or expression, so the shape parameters are only transmitted once, that is, the geometric shape of the head is fixed. In the subsequent transmission of image feature parameters, only posture parameters and expression parameters are transmitted. The first frame of the transmitted two-dimensional image can be used as a key frame image, or the key frame image can be selected according to the user's needs. Generally, the two-dimensional image of the front of the head is used as the key frame.

[0079] The embodiment of the present application provides a three-dimensional model reconstruction method, which obtains image feature parameters of the two-dimensional image by performing feature extraction on the collected two-dimensional image, and transmits the image feature parameters to the cloud instead of the reconstructed high-precision three-dimensional data of the human body model, which can greatly reduce the amount of data and effectively solve the problems of large amounts of calculations on the collection end and high data pressure on the transmission end.

[0080] In the embodiment provided in the present application, the parameters of the feature extraction layer and the reconstruction layer are adjusted in the following manner:

[0081] Input the sample image into the feature extraction layer to extract features of the sample image, and obtain an image feature vector corresponding to the image feature parameters;

[0082] Input the image feature vector to the reconstruction layer, use the basis function of the reconstruction layer to determine the weight parameters corresponding to different vertices in the reference 3D model based on the image feature vector, and calculate the coordinates of each vertex in the predicted 3D model according to the weight parameters, and determine and output the predicted 3D model based on the calculated coordinates of each vertex and the vertices of the triangle patch index;

[0083] The loss function value is determined according to the difference between the predicted three-dimensional model and the sample three-dimensional model, and the parameters of the feature extraction layer and the reconstruction layer are adjusted according to the loss function value.

[0084] First, the sample image is input into the feature extraction layer as shown in Formula 1 for feature extraction:

[0085]

[0086] in, represents the head shape parameters, represents the head posture parameters, represents the facial expression parameters, It can uniquely identify a vertex coordinate of the three-dimensional geometric model of the head. Bs() represents the influence function of the head shape parameter on the head model mesh T, Bp() represents the influence function of the head posture parameter on the head model mesh T, Be() represents the influence function of the facial expression parameter on the head model mesh T, S represents the weight of the head shape, P represents the weight of the head posture, and E represents the weight of the facial expression. Tp() represents the function of deforming the head model mesh T under the joint action of the head shape parameter, the head posture parameter and the facial expression parameter.

[0087] Then, the feature vector corresponding to each extracted image feature parameter is input into the reconstruction layer as shown in Formula 2 to reconstruct the three-dimensional model:

[0088]

[0089] Formula 2 can be regarded as a basis function, which includes two parts of standard linear blend skinning (LBS) and blend shape. The standard mesh model used has a mesh vertex number of N = 5023 and a joint number of K = 4 (located in the neck, jaw and two eyeballs). W represents the standard linear skinning function, which transforms the head model mesh T along the joint J. W is the skinning weight, which determines the weight parameter corresponding to different vertices in the reference 3D model.

[0090] The reference 3D model is composed of triangular facets and vertices indexed corresponding to each triangular facet. Each vertex corresponds to a unique coordinate. The vertices indexed corresponding to each triangular facet are fixed. When the head moves or the face produces an expression, the vertex coordinates will change.

[0091] Each vertex in the baseline 3D model corresponds to a weight parameter. For example, a smiling action is generally described as an upward movement of the corners of the mouth. Therefore, when using a sample image of a smile to train the model, the changes in the vertices of the mouth will be more obvious and have a larger weight, while the changes in the nose are not obvious and have a smaller weight. The coordinates of each vertex in the predicted 3D model are then calculated based on the weight parameters of each vertex, and the predicted 3D model is determined and output based on the calculated vertex coordinates and the vertices of the triangle patch index.

[0092] During the training process, the sample image is input into Formula 1, and the reconstructed sample 3D model is output after the above two steps of feature extraction and reconstruction. The loss function value is determined according to the difference between the predicted 3D model and the sample 3D model, and the parameters of the feature extraction layer and the reconstruction layer are adjusted according to the loss function value. Once the training is successful, as long as The above method can express a head model with real-time non-rigid deformation characteristics with a small number of parameters, which can be used as the basis for generating a three-dimensional head model from a single image and is not affected by the geometric loss of invisible areas.

[0093] Based on the same inventive concept, the present application embodiment provides a three-dimensional model reconstruction method, such as Figure 5 As shown, applied to the cloud, the method includes:

[0094] S501: receiving image feature parameters of a two-dimensional image transmitted by a collection end, wherein the image feature parameters include shape parameters, expression parameters and posture parameters.

[0095] Unlike the traditional method of receiving high-precision three-dimensional data of the reconstructed human body model, the amount of image feature parameter data of the two-dimensional image transmitted by the receiving acquisition end can be reduced by dozens of times compared with the traditional method. In this case, real-time three-dimensional communication can still be supported.

[0096] The process of extracting image feature parameters is as mentioned above.

[0097] S502: The image feature parameters are encoded and decoded and then transmitted to a rendering end.

[0098] The image feature parameters extracted from the two-dimensional image cannot be directly used to reconstruct the three-dimensional model on the rendering end, and need to be encoded and decoded in the cloud.

[0099] Based on the embodiment of the present application, a three-dimensional model reconstruction method is provided, such as Figure 6 As shown, applied to a rendering end, the method includes:

[0100] S601: receiving image feature parameters of a two-dimensional image transmitted from the cloud, wherein the image feature parameters include shape parameters, expression parameters, and posture parameters.

[0101] In the case where parameter instability occurs during the transmission of image feature parameters, the image feature parameters can be optimized in the cloud and the optimized image feature parameters can be transmitted to the rendering end.

[0102] Rendering devices may include but are not limited to: VR / AR helmets, TVs, and mobile phones.

[0103] The process of extracting the image feature parameters of the two-dimensional image is as described above.

[0104] S602: Input the image feature parameters into a three-dimensional reconstruction module, and use the three-dimensional reconstruction module to obtain a reconstructed three-dimensional model, wherein features of the sample image are extracted by inputting the sample image into a feature extraction layer, and the three-dimensional model is reconstructed and outputted according to the extracted image feature parameters by using a reconstruction layer, and parameters of the feature extraction layer and the reconstruction layer are adjusted with the goal of outputting the sample three-dimensional model, and at the end of the adjustment, the reconstruction layer is used as a three-dimensional reconstruction module.

[0105] The 3D reconstruction module is deployed on the rendering end. During training, a batch of samples can be used in the same device to implement the training of the feature extraction module and the 3D reconstruction module, and the samples can be input into a network model including a feature extraction layer and a reconstruction layer. The parameters of the feature extraction layer and the reconstruction layer are adjusted with the goal of outputting a sample 3D model. At the end of the adjustment, the reconstruction layer is used as the 3D reconstruction module.

[0106] The specific training process is as mentioned above.

[0107] The parameter transfer and optimization process of a three-dimensional model reconstruction method in the present application embodiment is as follows Figure 7 As shown, it includes the collection end, cloud end, and rendering end.

[0108] At the acquisition end: two-dimensional images are acquired through an RGBD camera, and then feature extraction module is used to extract features from the two-dimensional images to obtain image feature parameters, which include shape parameters, posture parameters and expression parameters. During the first transmission of image feature parameters, shape parameters, posture parameters and expression parameters are transmitted to the cloud, and then only posture parameters and expression parameters are transmitted.

[0109] In the cloud: Since the main transmission of posture parameters and expression parameters is done during the real-time parameter transmission process, the problem caused by this transmission process is that since these parameters are estimated for each frame, it may cause parameter instability, especially when the acquisition frame rate is not enough, the model of the rendering end is very likely to jump, especially the parts with more geometric triangles such as the mouth and eyes. Therefore, it is necessary to perform filtering optimization of the transmission parameters to weaken and reduce the rendering end model effect caused by the jump. At the same time, the real-time performance of the parameter optimization method must be considered, and it cannot affect the delay. After the image feature parameters are transmitted to the cloud, the image feature parameters are optimized.

[0110] In a possible implementation manner, encoding and decoding the image feature parameters and then transmitting them to the rendering end includes:

[0111] Using a high-frequency noise filtering algorithm to filter the same image feature parameter of at least one two-dimensional image, the high-frequency noise filtering algorithm includes sliding average, exponential sliding average method and SG filtering method;

[0112] The filtered image feature parameters are encoded and decoded and transmitted to the rendering end.

[0113] In a possible implementation manner, the parameter optimization process may also be performed before transmitting the image feature parameters to the cloud, wherein transmitting the image feature parameters to the cloud includes:

[0114] Using a high-frequency noise filtering algorithm to filter the same image feature parameter of at least one two-dimensional image, the high-frequency noise filtering algorithm includes sliding average, exponential sliding average method and SG filtering method;

[0115] The filtered image feature parameters are transmitted to the cloud.

[0116] The process of parameter transfer optimization is the filtering process, which can be performed before the image feature parameters are transmitted to the rendering end. After the image feature parameters are optimized in the cloud, the optimized image feature parameters are encoded and decoded and transmitted to the rendering end. The filtering process can filter out the high-frequency noise in the spectral data and better preserve the authenticity of the original signal. The jitter is eliminated, so that the rendering end can be reconstructed and rendered stably, improving the rendering effect.

[0117] At the rendering end, after receiving the image feature parameters, the head model is constructed using the image feature parameters, and finally the model is rendered.

[0118] Based on the same inventive concept, the embodiment of the present application provides a three-dimensional model reconstruction device 800, which is applied to a collection end, such as Figure 8 As shown, the device comprises:

[0119] The image acquisition module 801 is used to acquire a captured two-dimensional image and input the two-dimensional image into a feature extraction module, wherein the sample image is input into a feature extraction layer to extract features of the sample image, a reconstruction layer is used to reconstruct a three-dimensional model according to the extracted image feature parameters and output the model, and the parameters of the feature extraction layer and the reconstruction layer are adjusted with the output of the sample three-dimensional model as the goal, and the feature extraction layer is used as a feature extraction module after the adjustment is completed;

[0120] A feature extraction module 802 is used to extract image feature parameters of the two-dimensional image using the feature extraction module, wherein the image feature parameters include shape parameters, expression parameters and posture parameters;

[0121] The parameter transmission module 803 is used to transmit the image feature parameters to the cloud.

[0122] In a possible implementation, extracting image feature parameters of the two-dimensional image by using the feature extraction module includes:

[0123] The shape parameters, expression parameters and posture parameters of the two-dimensional image are extracted using the feature extraction module:

[0124] The parameter transmission module transmits the image feature parameters to the cloud, including:

[0125] When determining that the two-dimensional image is a key frame image, transmitting the shape parameters, expression parameters and posture parameters to the cloud;

[0126] When it is determined that the two-dimensional image is a non-key frame image, the expression parameters and the posture parameters are transmitted to the cloud.

[0127] In a possible implementation, the feature extraction module transmits the image feature parameters to the cloud, including:

[0128] Using a high-frequency noise filtering algorithm to filter the same image feature parameter of at least one two-dimensional image, the high-frequency noise filtering algorithm includes sliding average, exponential sliding average method and SG filtering method;

[0129] The filtered image feature parameters are transmitted to the cloud.

[0130] In a possible implementation, the parameters of the feature extraction layer and the reconstruction layer are adjusted in the following manner:

[0131] Input the sample image into the feature extraction layer to extract features of the sample image, and obtain an image feature vector corresponding to the image feature parameters;

[0132] Input the image feature vector to the reconstruction layer, use the basis function of the reconstruction layer to determine the weight parameters corresponding to different vertices in the reference 3D model based on the image feature vector, and calculate the coordinates of each vertex in the predicted 3D model according to the weight parameters, and determine and output the predicted 3D model based on the calculated coordinates of each vertex and the vertices of the triangle patch index;

[0133] The loss function value is determined according to the difference between the predicted three-dimensional model and the sample three-dimensional model, and the parameters of the feature extraction layer and the reconstruction layer are adjusted according to the loss function value.

[0134] Based on the embodiment of the present application, a three-dimensional model reconstruction device 900 is provided, which is applied to a collection end, such as Fig. 9 As shown, the device comprises:

[0135] A parameter receiving module 901 is used to receive image feature parameters of a two-dimensional image transmitted by a collection end, wherein the image feature parameters include shape parameters, expression parameters and posture parameters;

[0136] The parameter transmission module 902 is used to transmit the image feature parameters to the rendering end after encoding and decoding.

[0137] In a possible implementation manner, the parameter transmission module transmits the image feature parameters to the rendering end after encoding and decoding, including:

[0138] Using a high-frequency noise filtering algorithm to filter the same image feature parameter of at least one two-dimensional image, the high-frequency noise filtering algorithm includes sliding average, exponential sliding average method and SG filtering method;

[0139] The filtered image feature parameters are encoded and decoded and transmitted to the rendering end.

[0140] Based on the embodiment of the present application, a three-dimensional model reconstruction device 1000 is provided, which is applied to a rendering end, such as Fig.10 As shown, the device comprises:

[0141] A parameter receiving module 1001 is used to receive image feature parameters of a two-dimensional image transmitted from the cloud, wherein the image feature parameters include shape parameters, expression parameters and posture parameters;

[0142] The three-dimensional reconstruction module 1002 is used to input the image feature parameters into the three-dimensional reconstruction module, and use the three-dimensional reconstruction module to obtain a reconstructed three-dimensional model, wherein the sample image is input into the feature extraction layer to extract features of the sample image, and the reconstruction layer is used to reconstruct and output the three-dimensional model according to the extracted image feature parameters, and the parameters of the feature extraction layer and the reconstruction layer are adjusted with the goal of outputting the sample three-dimensional model. When the adjustment is completed, the reconstruction layer is used as the three-dimensional reconstruction module.

[0143] Based on the same inventive concept, the present application provides a three-dimensional model reconstruction device, such as Fig.11 As shown, the device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the three-dimensional model reconstruction methods in the above-mentioned embodiments.

[0144] Refer to the following Fig.11 The electronic device 130 according to this embodiment of the present application is described. Fig.11 The electronic device 130 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0145] like Fig.11 As shown, the electronic device 130 is in the form of a general electronic device. The components of the electronic device 130 may include but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).

[0146] The processor 131 is used to read and execute instructions in the memory 132, so that the at least one processor can execute the method for reconstructing a three-dimensional model provided in the above embodiment.

[0147] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0148] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0149] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0150] The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 130, and / or communicate with any device that enables the electronic device 130 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 135. Furthermore, the electronic device 130 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 136. As shown, the network adapter 136 communicates with other modules for the electronic device 130 via a bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0151] In some possible implementations, various aspects of a three-dimensional model reconstruction method provided by the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a three-dimensional model reconstruction method according to various exemplary embodiments of the present application described above in this specification.

[0152] In addition, the present application also provides a computer program product, which is used to enable a computer to execute any of the methods described in the above embodiments.

[0153] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0155] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0156] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A three-dimensional model reconstruction method, characterized in that: Applied to the acquisition end, the method includes: The collected two-dimensional image is obtained, and the two-dimensional image is input into a feature extraction module, wherein the input feature extraction layer is used to perform feature extraction on the sample image to obtain an image feature vector corresponding to the sample image, and the basis function of the reconstruction layer is used to determine the weight parameters corresponding to different vertices in the reference three-dimensional model based on the image feature vector, and the coordinates of each vertex in the predicted three-dimensional model are obtained according to the weight parameters, and the predicted three-dimensional model is determined based on the coordinates of each vertex and the vertices of the triangle patch index, and the loss function value is determined according to the difference between the predicted three-dimensional model and the sample three-dimensional model, and the parameters of the feature extraction layer and the reconstruction layer are adjusted according to the loss function value. When the adjustment is completed, the feature extraction layer is used as the feature extraction module, and the basis function is M(β → ,θ → ,ψ → )=W(T P (β → ,θ → ,ψ → ),J(β → ),θ → ,w), w is the skin weight, and the formula of the feature extraction layer is: T P (β → ,θ → ,ψ → )=T+B S (β → ; S)+B P (θ → ;P)+B E (ψ → ; E), W represents the standard linear skinning function, which transforms the head model mesh T along the joint J, β → represents the head shape parameter, θ → represents the head posture parameter, ψ → represents the facial expression parameter, B S () represents the influence function of head shape parameters on the head model mesh T, B P () represents the influence function of head posture parameters on the head model grid T, B E () represents the influence function of facial expression parameters on the head model mesh T, S represents the weight of the head shape, P represents the weight of the head posture, E represents the weight of the facial expression, T P () represents a function for deforming the head model mesh T under the joint action of head shape parameters, head posture parameters and facial expression parameters; Extracting image feature parameters of the two-dimensional image using the feature extraction module, wherein the image feature parameters include shape parameters, expression parameters and posture parameters; The shape parameters, expression parameters and posture parameters of the two-dimensional image determined as the key frame image are transmitted to the cloud, and the expression parameters and posture parameters of the two-dimensional image determined as the non-key frame image are transmitted to the cloud, so that the cloud encodes and decodes the shape parameters, expression parameters and posture parameters and transmits them to the rendering end. The key frame image is the two-dimensional image transmitted during the first image feature parameter transmission process.

2. The method according to claim 1, characterized in that Transmitting the image feature parameters to the cloud includes: Using a high-frequency noise filtering algorithm to filter the same image feature parameter of at least one two-dimensional image, the high-frequency noise filtering algorithm includes sliding average, exponential sliding average method and SG filtering method; The filtered image feature parameters are transmitted to the cloud.

3. A three-dimensional model reconstruction method, characterized in that: Applied to the cloud, the method includes: The shape parameters, expression parameters and posture parameters of a two-dimensional image determined as a key frame image transmitted by the acquisition end, as well as the expression parameters and posture parameters of a two-dimensional image determined as a non-key frame image, are received. The key frame image is a two-dimensional image transmitted during the first image feature parameter transmission process of the acquisition end. The shape parameters, expression parameters and posture parameters are extracted by a feature extraction module, wherein a sample image is subjected to feature extraction using an input feature extraction layer to obtain an image feature vector corresponding to the sample image, a weight parameter corresponding to different vertices in a reference three-dimensional model is determined based on the image feature vector using a basis function of a reconstruction layer, and the coordinates of each vertex in a predicted three-dimensional model are obtained based on the weight parameter, a predicted three-dimensional model is determined based on the coordinates of each vertex and the vertices of a triangular face index, a loss function value is determined based on the difference between the predicted three-dimensional model and the sample three-dimensional model, and parameters of the feature extraction layer and the reconstruction layer are adjusted based on the loss function value. When the adjustment is completed, the feature extraction layer is used as a feature extraction module, and the basis function is M(β → ,θ → ,ψ → )=W(T P (β → ,θ → ,ψ → ),J(β → ),θ → ,w), w is the skin weight, and the formula of the feature extraction layer is: T P (β → ,θ → ,ψ → )=T+B S (β → ; S)+B P (θ → ;P)+B E (ψ → ; E), W represents the standard linear skinning function, which transforms the head model mesh T along the joint J, β → represents the head shape parameter, θ → represents the head posture parameter, ψ → represents the facial expression parameter, B S () represents the influence function of head shape parameters on the head model mesh T, B P () represents the influence function of head posture parameters on the head model grid T, B E () represents the influence function of facial expression parameters on the head model mesh T, S represents the weight of the head shape, P represents the weight of the head posture, E represents the weight of the facial expression, T P () represents a function for deforming the head model mesh T under the joint action of head shape parameters, head posture parameters and facial expression parameters; The shape parameters, expression parameters and posture parameters are encoded and decoded and then transmitted to the rendering end.

4. The method according to claim 3, characterized in that The image feature parameters are encoded and decoded and then transmitted to the rendering end, including: Using a high-frequency noise filtering algorithm to filter the same image feature parameter of at least one two-dimensional image, the high-frequency noise filtering algorithm includes sliding average, exponential sliding average method and SG filtering method; The filtered image feature parameters are encoded and decoded and transmitted to the rendering end.

5. A three-dimensional model reconstruction method, characterized in that: Applied to the rendering end, the method includes: Receiving shape parameters, expression parameters and posture parameters of key frame images transmitted from the cloud, as well as expression parameters and posture parameters of non-key frame images, wherein the key frame images are two-dimensional images transmitted during the first image feature parameter transmission process of the acquisition end; The image feature parameters are input into a 3D reconstruction module, and a reconstructed 3D model is obtained by using the 3D reconstruction module, wherein a sample image is subjected to feature extraction by using an input feature extraction layer to obtain an image feature vector corresponding to the sample image, and a basis function of a reconstruction layer is used to determine weight parameters corresponding to different vertices in a reference 3D model based on the image feature vector, and the coordinates of each vertex in a predicted 3D model are obtained according to the weight parameters, and a predicted 3D model is determined based on the coordinates of each vertex and the vertices of a triangular patch index, and a loss function value is determined according to a difference between the predicted 3D model and the sample 3D model, and parameters of the feature extraction layer and the reconstruction layer are adjusted according to the loss function value, and at the end of the adjustment, the reconstruction layer is used as a 3D reconstruction module, and the basis function is M(β → ,θ → ,ψ → )=W(T P (β → ,θ → ,ψ → ),J(β → ),θ → ,w), w is the skin weight, and the formula of the feature extraction layer is: T P (β → ,θ → ,ψ → )=T+B S (β → ; S)+B P (θ → ;P)+B E (ψ → ; E), W represents the standard linear skinning function, which transforms the head model mesh T along the joint J, β → represents the head shape parameter, θ → represents the head posture parameter, ψ → represents the facial expression parameter, B S () represents the influence function of head shape parameters on the head model mesh T, B P () represents the influence function of head posture parameters on the head model grid T, B E () represents the influence function of facial expression parameters on the head model mesh T, S represents the weight of the head shape, P represents the weight of the head posture, E represents the weight of the facial expression, T P () represents the function that deforms the head model mesh T under the joint action of head shape parameters, head posture parameters and facial expression parameters.

6. A three-dimensional model reconstruction device, characterized in that: Applied to the collection end, the device includes: The image acquisition module is used to obtain the acquired two-dimensional image and input the two-dimensional image into the feature extraction module, wherein the input feature extraction layer is used to perform feature extraction on the sample image to obtain the image feature vector corresponding to the sample image, the basis function of the reconstruction layer is used to determine the weight parameters corresponding to different vertices in the reference three-dimensional model based on the image feature vector, and the coordinates of each vertex in the predicted three-dimensional model are obtained according to the weight parameters, the predicted three-dimensional model is determined based on the coordinates of each vertex and the vertices of the triangle patch index, the loss function value is determined according to the difference between the predicted three-dimensional model and the sample three-dimensional model, the parameters of the feature extraction layer and the reconstruction layer are adjusted according to the loss function value, and the feature extraction layer is used as the feature extraction module when the adjustment is completed, and the basis function is M(β → ,θ → ,ψ → )=W(T P (β → ,θ → ,ψ → ),J(β → ),θ → ,w), w is the skin weight, and the formula of the feature extraction layer is: T P (β → ,θ → ,ψ → )=T+B S (β → ; S)+B P (θ → ;P)+B E (ψ → ; E), W represents the standard linear skinning function, which transforms the head model mesh T along the joint J, β → represents the head shape parameter, θ → represents the head posture parameter, ψ → represents the facial expression parameter, B S () represents the influence function of head shape parameters on the head model mesh T, B P () represents the influence function of head posture parameters on the head model grid T, B E () represents the influence function of facial expression parameters on the head model mesh T, S represents the weight of the head shape, P represents the weight of the head posture, E represents the weight of the facial expression, T P () represents a function for deforming the head model mesh T under the joint action of head shape parameters, head posture parameters and facial expression parameters; A feature extraction module, used to extract image feature parameters of the two-dimensional image using the feature extraction module, wherein the image feature parameters include shape parameters, expression parameters and posture parameters; A parameter transmission module is used to transmit the shape parameters, expression parameters and posture parameters of a two-dimensional image determined as a key frame image to the cloud, and to transmit the expression parameters and posture parameters of a two-dimensional image determined as a non-key frame image to the cloud, so that the cloud can encode and decode the shape parameters, expression parameters and posture parameters and then transmit them to the rendering end. The key frame image is the two-dimensional image transmitted during the first image feature parameter transmission process.

7. A three-dimensional model reconstruction device, characterized in that: Applied to the cloud, the device comprises: A parameter receiving module is used to receive shape parameters, expression parameters and posture parameters of a two-dimensional image determined as a key frame image transmitted by the acquisition end and transmitted to the cloud, as well as expression parameters and posture parameters of a two-dimensional image determined as a non-key frame image, wherein the key frame image is a two-dimensional image transmitted during the first image feature parameter transmission of the acquisition end, and the shape parameters, expression parameters and posture parameters are extracted by a feature extraction module, wherein the sample image is subjected to feature extraction using an input feature extraction layer to obtain an image feature vector corresponding to the sample image, and the weight parameters corresponding to different vertices in the reference three-dimensional model are determined based on the image feature vector using the basis function of the reconstruction layer, and the coordinates of each vertex in the predicted three-dimensional model are obtained based on the weight parameters, and the predicted three-dimensional model is determined based on the coordinates of each vertex and the vertices of the triangular face index, and the loss function value is determined based on the difference between the predicted three-dimensional model and the sample three-dimensional model, and the parameters of the feature extraction layer and the reconstruction layer are adjusted based on the loss function value, and when the adjustment is completed, the feature extraction layer is used as a feature extraction module, and the basis function is M(β → ,θ → ,ψ → )=W(T P (β → ,θ → ,ψ → ),J(β → ),θ → ,w), w is the skin weight, and the formula of the feature extraction layer is: T P (β → ,θ → ,ψ → )=T+B S (β → ; S)+B P (θ → ;P)+B E (ψ → ; E), W represents the standard linear skinning function, which transforms the head model mesh T along the joint J, β → represents the head shape parameter, θ → represents the head posture parameter, ψ → represents the facial expression parameter, B S () represents the influence function of head shape parameters on the head model mesh T, B P () represents the influence function of head posture parameters on the head model grid T, B E () represents the influence function of facial expression parameters on the head model mesh T, S represents the weight of the head shape, P represents the weight of the head posture, E represents the weight of the facial expression, T P () represents a function for deforming the head model mesh T under the joint action of head shape parameters, head posture parameters and facial expression parameters; The parameter transmission module is used to encode and decode the shape parameters, expression parameters and posture parameters and transmit them to the rendering end.

8. A three-dimensional model reconstruction device, characterized in that: The device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute the method as described in any one of claims 1-2, or execute the method as described in any one of claims 3-4, or execute the method as described in claim 5.

Images