Object reconstruction method, device, electronic device and storage medium
By obtaining the image and lighting information of the target object from multiple perspectives, generating features and updating the three-dimensional reconstruction information in combination with shape characterization network and image rendering network, the problems of low resolution of the three-dimensional reconstruction information and imbalance in the existing technology are solved. Achieve more accurate and continuous object shape representation and virtual image generation.
Patent Information
- Application Number
- CN202111199719.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-14
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-10-14
AI Technical Summary
In the prior art, the three-dimensional reconstruction information resolution is low and the problem of the object reconstruction information resolution and storage overhead cannot be balanced.
By obtaining the image and lighting information of the target object from multiple perspectives, generating object features, color features and differential attribute features, combining shape representation networks and image rendering networks for object shape representation and rendering, updating three-dimensional reconstruction information until the preset conditions are met .
It improves the accuracy of object shape representation and the continuity of three-dimensional reconstruction information, reduces storage overhead, meets the needs of multi-resolution applications, and improves the vividness and resolution of virtual images.
Smart Images

Figure CN113902848B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer vision technology, and in particular to an object reconstruction method, device, electronic device and storage medium. Background Art
[0002] With the development of computer vision technology, image processing technology is being used in more and more scenarios for virtual images, such as games, social networking, photography, etc. In order to improve the realism of virtual images, it is often necessary to perform three-dimensional reconstruction of objects required to construct virtual images, such as faces and hands.
[0003] In the related art, the reconstruction of objects such as faces usually uses traditional triangles (triangles, meshes) to represent the three-dimensional shape of the object. Specifically, based on 3D Morphable Models (3DMMs), a deep learning codec network is used to extract features from the input multi-view images, so as to learn the coefficients of each group of basic shapes of 3DMMs; finally, the weighted average shape blending is used to achieve the reconstruction of the three-dimensional face shape. However, the shape representation based on triangles in the above-mentioned related art is discontinuous and cannot fully represent the shape of the object, resulting in a low resolution of the three-dimensional reconstruction information of the related art. In addition, the storage overhead of the shape representation based on triangles is positively correlated with the resolution. The higher the resolution, the more storage space is required, and there is a problem of being unable to balance the resolution of the object reconstruction information and the storage overhead. Summary of the invention
[0004] The present disclosure provides an object reconstruction method, device, electronic device and storage medium to at least solve the problem of low resolution of three-dimensional reconstruction information and the inability to balance the resolution of object reconstruction information and storage overhead in the related art. The technical solution of the present disclosure is as follows:
[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an object reconstruction method, comprising:
[0006] Acquire multiple target object images of the target object captured from multiple viewing angles and illumination information of each pixel in the multiple target object images;
[0007] generating an object feature of the target object, a color feature of the target object, and a plurality of object difference attribute features corresponding to the target object under the multiple viewing angles;
[0008] Inputting the pixel coordinates of each pixel point, the object feature and the plurality of object difference attribute features into a shape representation network to perform object shape representation, thereby obtaining object shape representation information, wherein the object shape representation information represents the distance information from each pixel point to the geometric shape boundary of the target object;
[0009] Inputting the pixel coordinates of each pixel point, the object shape representation information, the color feature and the illumination information into an image rendering network for image rendering processing to obtain a plurality of rendered images under the multi-viewing angles;
[0010] Based on the multiple rendering images and the multiple target object images, the object shape representation information is updated by object reconstruction to obtain three-dimensional reconstruction information of the target object.
[0011] Optionally, the performing object reconstruction and updating on the object shape representation information based on the multiple rendered images and the multiple target object images to obtain the three-dimensional reconstruction information of the target object includes:
[0012] determining a first target loss based on the plurality of rendered images and the plurality of target object images;
[0013] When the first target loss does not satisfy a first preset condition, updating the object feature, the plurality of object difference attribute features and the color feature;
[0014] Based on the updated object feature, the updated multiple object difference attribute features and the updated color feature, repeating the step of inputting the pixel coordinates of each pixel point, the object feature and the multiple object difference attribute features into the shape representation network to perform object shape representation to obtain object shape representation information, to the step of determining the first target loss according to the multiple rendered images and the multiple target object images, until the first target loss meets the first preset condition;
[0015] When the first target loss satisfies the first preset condition, the object shape representation information output by the current shape representation network is used as the three-dimensional reconstruction information.
[0016] Optionally, the method further includes:
[0017] Determining pixel coordinates of each pixel in the plurality of target object images;
[0018] Encoding the pixel coordinates to obtain pixel coordinate features;
[0019] The step of inputting the pixel coordinates of each pixel point, the object features and the plurality of object difference attribute features into a shape representation network to perform object shape representation to obtain object shape representation information comprises:
[0020] Inputting the pixel coordinate feature, the object feature and the plurality of object difference attribute features into the shape representation network to perform object shape representation, thereby obtaining the object shape representation information;
[0021] The step of inputting the pixel coordinates of each pixel point, the object shape representation information, the color feature and the illumination information into an image rendering network for image rendering processing to obtain the plurality of rendered images under the multi-viewing angles comprises:
[0022] The pixel coordinate features, the object shape representation information, the color features and the illumination information are input into the image rendering network for image rendering processing to obtain the multiple rendered images.
[0023] Optionally, generating the object feature of the target object, the color feature of the target object, and a plurality of object difference attribute features corresponding to the target object under the multiple viewing angles includes:
[0024] Based on the first preset dimension information, randomly generate the object feature;
[0025] Based on the second preset dimension information, randomly generate the color feature;
[0026] Based on the third preset dimension information, the plurality of object difference attribute features are randomly generated.
[0027] Optionally, the method further includes:
[0028] Acquire a plurality of sample object images of the sample object collected under the multiple viewing angles and sample illumination information of each sample pixel point in the plurality of sample object images;
[0029] generating a sample object feature of the sample object, a sample color feature of the sample object, and a plurality of sample object difference attribute features corresponding to the sample object under the multiple viewing angles;
[0030] Inputting the pixel coordinates of each sample pixel point, the sample object feature and the plurality of sample object difference attribute features into a first preset neural network to perform object shape characterization, and obtaining sample object shape characterization information, wherein the sample object shape characterization information is distance information from each sample pixel point to a geometric shape boundary of the sample object;
[0031] Inputting the pixel coordinates of each sample pixel point, the sample object shape representation information, the sample color feature and the sample illumination information into a second preset neural network for image rendering processing to obtain a plurality of sample rendering images under the multi-viewing angles;
[0032] Based on the multiple sample rendering images and the multiple sample object images, the first preset neural network and the second preset neural network are trained to obtain the shape representation network and the image rendering network.
[0033] Optionally, the training the first preset neural network and the second preset neural network based on the multiple sample rendering images and the multiple sample object images to obtain the shape representation network and the image rendering network includes:
[0034] determining a second target loss based on the plurality of sample rendered images and the plurality of sample object images;
[0035] When the second target loss does not satisfy a second preset condition, updating the sample object feature, the plurality of sample object difference attribute features, the sample color feature, a first network parameter of the first preset neural network, and a second network parameter of the second preset neural network;
[0036] Based on the updated sample object feature, the updated multiple sample object difference attribute features, the updated sample color feature, the updated first preset neural network and the updated second preset neural network, repeating the step of inputting the pixel coordinates of each sample pixel point, the sample object feature and the multiple sample object difference attribute features into the first preset neural network for object shape characterization to obtain sample object shape characterization information, to the step of determining the second target loss according to the multiple sample rendered images and the multiple sample object images, until the second target loss meets the second preset condition;
[0037] When the second target loss satisfies the second preset condition, the current first preset neural network is used as the shape representation network, and the current second preset neural network is used as the image rendering network.
[0038] Optionally, the method further includes:
[0039] Obtaining the three-dimensional coordinates corresponding to each point on the three-dimensional model of the preset object, the labeled object shape representation information of the preset object, and the labeled normal vector corresponding to each point;
[0040] Inputting the three-dimensional coordinates into an initial neural network to characterize the object shape, and obtaining a predicted normal vector and predicted object shape characterization information corresponding to the preset object;
[0041] Determining a third target loss according to the predicted normal vector, the predicted object shape representation information, the labeled object shape representation information, and the labeled normal vector;
[0042] The initial neural network is trained based on the third target loss to obtain the first preset neural network.
[0043] According to a second aspect of an embodiment of the present disclosure, there is provided an object reconstruction device, comprising:
[0044] An information acquisition module is configured to acquire a plurality of target object images of a target object captured from multiple viewing angles and illumination information of each pixel in the plurality of target object images;
[0045] A feature generation module is configured to generate an object feature of the target object, a color feature of the target object, and a plurality of object difference attribute features corresponding to the target object under the multiple viewing angles;
[0046] A first object shape characterization module is configured to input the pixel coordinates of each pixel point, the object feature and the plurality of object difference attribute features into a shape characterization network to perform object shape characterization, and obtain object shape characterization information, wherein the object shape characterization information represents the distance information from each pixel point to the geometric shape boundary of the target object;
[0047] A first image rendering processing module is configured to input the pixel coordinates of each pixel point, the object shape representation information, the color feature and the illumination information into an image rendering network for image rendering processing to obtain a plurality of rendered images under the multi-viewing angles;
[0048] The object reconstruction update module is configured to perform object reconstruction update on the object shape representation information based on the multiple rendering images and the multiple target object images to obtain three-dimensional reconstruction information of the target object.
[0049] Optionally, the object reconstruction update module includes:
[0050] A first target loss determining unit is configured to determine a first target loss based on the plurality of rendered images and the plurality of target object images;
[0051] a feature updating unit, configured to update the object feature, the plurality of object difference attribute features and the color feature if the first target loss does not satisfy a first preset condition;
[0052] A first iteration unit is configured to execute, based on the updated object feature, the updated multiple object difference attribute features and the updated color feature, repeating the steps of inputting the pixel coordinates of each pixel point, the object feature and the multiple object difference attribute features into a shape representation network to perform object shape representation to obtain object shape representation information, to the step of determining a first target loss according to the multiple rendered images and the multiple target object images, until the first target loss meets the first preset condition;
[0053] The three-dimensional reconstruction information determination unit is configured to execute, when the first target loss satisfies the first preset condition, using the object shape representation information output by the current shape representation network as the three-dimensional reconstruction information.
[0054] Optionally, the device further comprises:
[0055] A pixel coordinate determination module is configured to determine the pixel coordinates of each pixel point in the plurality of target object images;
[0056] An encoding processing module is configured to perform encoding processing on the pixel coordinates to obtain pixel coordinate features;
[0057] The first object shape characterization module is further configured to input the pixel coordinate feature, the object feature and the plurality of object difference attribute features into the shape characterization network to perform object shape characterization to obtain the object shape characterization information;
[0058] The first image rendering processing module is further configured to input the pixel coordinate features, the object shape representation information, the color features and the lighting information into the image rendering network for image rendering processing to obtain the multiple rendered images.
[0059] Optionally, the feature generation module includes:
[0060] An object feature generating unit, configured to randomly generate the object feature based on the first preset dimension information;
[0061] A color feature generating unit is configured to randomly generate the color feature based on the second preset dimension information;
[0062] The object difference attribute feature generating unit is configured to randomly generate the plurality of object difference attribute features based on the third preset dimension information.
[0063] Optionally, the device further comprises:
[0064] A first sample information acquisition module is configured to acquire a plurality of sample object images of the sample object collected under the multiple viewing angles and sample illumination information of each sample pixel in the plurality of sample object images;
[0065] A sample feature generation module is configured to generate a sample object feature of the sample object, a sample color feature of the sample object, and a plurality of sample object difference attribute features corresponding to the sample object under the multiple viewing angles;
[0066] A second object shape characterization module is configured to input the pixel coordinates of each sample pixel point, the sample object feature, and the plurality of sample object difference attribute features into a first preset neural network to perform object shape characterization, and obtain sample object shape characterization information, wherein the sample object shape characterization information is distance information from each sample pixel point to a geometric shape boundary of the sample object;
[0067] A second image rendering processing module is configured to execute image rendering processing by inputting the pixel coordinates of each sample pixel point, the sample object shape representation information, the sample color feature and the sample illumination information into a second preset neural network to obtain a plurality of sample rendering images under the multi-viewing angles;
[0068] The first network training module is configured to train the first preset neural network and the second preset neural network based on the multiple sample rendering images and the multiple sample object images to obtain the shape representation network and the image rendering network.
[0069] Optionally, the first network training module includes:
[0070] A second target loss determining unit is configured to determine a second target loss based on the plurality of sample rendered images and the plurality of sample object images;
[0071] a training data updating unit, configured to update the sample object feature, the plurality of sample object difference attribute features, the sample color feature, a first network parameter of the first preset neural network, and a second network parameter of the second preset neural network when the second target loss does not satisfy a second preset condition;
[0072] The second iteration unit is configured to execute the steps of inputting the pixel coordinates of each sample pixel point, the sample object feature and the multiple sample object difference attribute features into the first preset neural network to characterize the object shape and obtain the sample object shape characterization information, and then determining the second target loss according to the multiple sample rendered images and the multiple sample object images based on the updated sample object feature, the updated multiple sample object difference attribute features, the updated sample color feature, the updated first preset neural network and the updated second preset neural network, until the second target loss meets the second preset condition;
[0073] The network determination unit is configured to execute, when the second target loss satisfies the second preset condition, using the current first preset neural network as the shape characterization network and using the current second preset neural network as the image rendering network.
[0074] Optionally, the device further comprises:
[0075] A second sample information acquisition module is configured to execute acquisition of the three-dimensional coordinates corresponding to each point on the three-dimensional model of the preset object, the labeled object shape representation information of the preset object, and the labeled normal vector corresponding to each point;
[0076] A third object shape characterization module is configured to execute the input of the three-dimensional coordinates into the initial neural network to perform object shape characterization, and obtain a predicted normal vector and predicted object shape characterization information corresponding to the preset object;
[0077] A third target loss determination module is configured to determine a third target loss based on the predicted normal vector, the predicted object shape representation information, the labeled object shape representation information and the labeled normal vector;
[0078] The second network training module is configured to perform training of the initial neural network based on the third target loss to obtain the first preset neural network.
[0079] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a method as described in any one of the above-mentioned first aspects.
[0080] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any method described in the first aspect of the embodiment of the present disclosure.
[0081] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any one of the methods described in the first aspect of the embodiment of the present disclosure.
[0082] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0083] The pixel coordinates of the pixels in multiple target object images under multiple perspectives, the object features and the multiple object difference attribute features under multiple perspectives are combined to perform object shape representation learning, and the learned object shape representation information is rendered in combination with the image rendering network. The rendered image and the target object image can be combined to reconstruct and update the object shape representation information, thereby improving the accuracy of the shape representation network's shape representation of the target object. The object is reconstructed in combination with the object shape representation that represents the distance information from each pixel to the geometric shape boundary of the target object. This can effectively reduce the storage overhead of the object representation while improving the continuity of the reconstructed 3D reconstruction information, thereby achieving a more comprehensive and accurate object shape representation. While improving the resolution of the object reconstruction information, it can meet the application requirements of multiple resolutions, thereby improving the vividness and resolution of the subsequent virtual image generated based on the 3D reconstruction information.
[0084] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0085] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0086] Figure 1 is a schematic diagram of an application environment according to an exemplary embodiment;
[0087] Figure 2 is a flow chart of an object reconstruction method according to an exemplary embodiment;
[0088] Figure 3 is a flowchart of a training shape representation network and an image rendering network according to an exemplary embodiment;
[0089] Figure 4 It is a flowchart showing a method of training a first preset neural network and a second preset neural network based on a plurality of sample rendering images and a plurality of sample object images to obtain a shape representation network and an image rendering network according to an exemplary embodiment;
[0090] Figure 5 is a flow chart showing a method of generating a first preset neural network according to an exemplary embodiment;
[0091] Figure 6 A flowchart of performing object reconstruction and updating on object shape representation information based on multiple rendering images and multiple target object images to obtain three-dimensional reconstruction information of the target object is shown according to an exemplary embodiment;
[0092] Figure 7 is a flow chart showing another object reconstruction method according to an exemplary embodiment;
[0093] Figure 8 is a block diagram of an object reconstruction device according to an exemplary embodiment;
[0094] Fig. 9 is a block diagram of an electronic device for object reconstruction according to an exemplary embodiment;
[0095] Fig.10 It is a block diagram of an electronic device for object reconstruction according to an exemplary embodiment. DETAILED DESCRIPTION
[0096] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0097] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0098] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.
[0099] See also Figure 1 , Figure 1 is a schematic diagram of an application environment according to an exemplary embodiment. Figure 1 As shown, the application environment may include a server 100 and a terminal 200 .
[0100] In an optional embodiment, the server 100 can be used to train the shape representation network and the image rendering network. Specifically, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.
[0101] In an optional embodiment, the terminal 200 can be used to reconstruct the shape of the target object based on the shape representation network and image rendering network trained by the server 100 to obtain three-dimensional reconstruction information of the target object. Specifically, the terminal 200 may include but is not limited to electronic devices such as smart phones, desktop computers, tablet computers, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, smart wearable devices, etc., and may also be software running on the above electronic devices, such as applications, etc. Optionally, the operating system running on the electronic device may include but is not limited to Android system, IOS system, Linux, Windows, etc.
[0102] In addition, it should be noted that Figure 1 What is shown is only one application environment provided by the present disclosure. In practical applications, other application environments may also be included. For example, the shape reconstruction of the target object based on the trained shape representation network and image rendering network may also be implemented on the server side.
[0103] In the embodiments of this specification, the server 100 and the terminal 200 may be directly or indirectly connected via wired or wireless communication, which is not limited in this disclosure.
[0104] Figure 2 is a flowchart of an object reconstruction method according to an exemplary embodiment. Figure 2 As shown, the object reconstruction method is used in electronic devices such as terminals and servers, and includes the following steps.
[0105] In step S201, a plurality of target object images of a target object captured from multiple viewing angles and illumination information of each pixel in the plurality of target object images are obtained.
[0106] In an optional embodiment, the target object may be an object that needs to be 3D reconstructed, which may be set in combination with the actual application scenario requirements. Optionally, in the application scenario of expression production, the target object may include a face. Optionally, in the application scenario of character animation production, the target object may include elements required in the character animation production process such as hands and feet.
[0107] In an optional embodiment, the above-mentioned multiple target object images can be object images under multiple perspectives (i.e., multiple perspectives) obtained by rendering a three-dimensional model of the target object, or can be images of the target object taken from multiple perspectives by a camera device. Specifically, the selection of multiple perspectives can be set in combination with actual applications, such as the front, back, and both sides of the target object.
[0108] In a specific embodiment, the above-mentioned illumination information may include an incident vector and a normal vector corresponding to each pixel point. Specifically, the incident vector corresponding to each pixel point may be the direction information of the incident light received by the three-dimensional point (a point on the target object or a point on the three-dimensional model) corresponding to the pixel point when the target object image at the corresponding viewing angle is collected; the normal vector corresponding to each pixel point may be the normal vector of the surface where the three-dimensional point corresponding to the pixel point is located when the target object image at the corresponding viewing angle is collected.
[0109] In step S203, an object feature of the target object, a color feature of the target object, and a plurality of object difference attribute features corresponding to the target object under multiple viewing angles are generated.
[0110] In a specific embodiment, the object feature may be the common feature information of the target object under multiple viewing angles, and the color feature of the target object may be the feature information characterizing the color of the target object; multiple target object images include the same target object, and accordingly, the multiple target object images correspond to the same object feature. Specifically, multiple object difference attribute features may characterize the target attributes of the target object under multiple viewing angles, and specifically, the target attribute may be an attribute in the attribute information of the target object that may differ due to different viewing angles. In a specific embodiment, taking the target object as a face as an example, the target attribute may be an expression; taking the target object as a hand as an example, the target attribute may be a gesture.
[0111] In a specific embodiment, the dimension information of the object feature, the color feature, and the multiple object difference attribute features may be predetermined in combination with the actual application situation. Optionally, the above-mentioned object feature of the target object, the color feature of the target object, and the multiple object difference attribute features corresponding to the target object under multiple viewing angles may include:
[0112] Based on the first preset dimension information, randomly generate object features;
[0113] Based on the second preset dimension information, randomly generate color features;
[0114] Based on the third preset dimension information, a plurality of object difference attribute features are randomly generated.
[0115] In a specific embodiment, the first preset dimension information may be dimension information for presetting object features; the second preset dimension information may be dimension information for presetting color features; and the third preset dimension information may be dimension information for presetting multiple object difference attribute features.
[0116] In the above embodiment, the object features, color features and multiple object difference attribute features under multiple perspectives of the target object are randomly generated in combination with the preset dimensional information, which can facilitate the subsequent learning of more comprehensive object shape information and improve the accuracy of object reconstruction.
[0117] In step S205, the pixel coordinates, object features and multiple object difference attribute features of each pixel are input into a shape representation network to perform object shape representation, thereby obtaining object shape representation information.
[0118] In a specific embodiment, the object shape representation information may represent the distance information from each pixel to the geometric shape boundary of the target object. In a specific embodiment, the distance information may include a distance value and a sign, wherein the sign includes a positive sign and a negative sign; a positive sign represents that the corresponding pixel is inside the geometric shape boundary of the target object; a negative sign represents that the corresponding pixel is outside the geometric shape boundary of the target object.
[0119] In a specific embodiment, the object shape representation information may be a SDF (Signed Distance Function) value. Specifically, the SDF value may be Among them, f() can represent the signed distance function, and [·,·,·] can represent the pair z1 and z2 are merged. It can represent the pixel coordinates of pixel points in multiple target object images, z1 can represent object features, and z2 can represent multiple object difference attribute features.
[0120] In a specific embodiment, the shape representation network can be pre-trained, and the shape representation network can be used to learn the shape representation of different objects. Optionally, in the process of training the shape representation network, in order to improve the accuracy of the shape representation network for different objects, the image rendering network can be trained in the process of training the shape representation network in combination with the object shape representation information obtained by the shape representation network. Specifically, the image rendering network can be used to render the object shape representation information corresponding to the image, and then the rendered image output by the image rendering network can be combined to achieve self-supervised training of the shape representation network.
[0121] Accordingly, the above method may further include the steps of training a shape representation network and an image rendering network, specifically, Figure 3 As shown, the following steps may be included:
[0122] In step S301, a plurality of sample object images of the sample object collected under multiple viewing angles and sample illumination information of each sample pixel point in the plurality of sample object images are obtained;
[0123] In step S303, a sample object feature of the sample object, a sample color feature of the sample object, and a plurality of sample object difference attribute features corresponding to the sample object under multiple viewing angles are generated;
[0124] In step S305, the pixel coordinates of each sample pixel point, the sample object features and the multiple sample object difference attribute features are input into a first preset neural network to perform object shape characterization to obtain sample object shape characterization information;
[0125] In step S307, the pixel coordinates of each sample pixel point, the sample object shape representation information, the sample color characteristics and the sample illumination information are input into a second preset neural network for image rendering processing to obtain a plurality of sample rendering images under multiple viewing angles;
[0126] In step S309, based on the multiple sample rendering images and the multiple sample object images, the first preset neural network and the second preset neural network are trained to obtain a shape representation network and an image rendering network.
[0127] In a specific embodiment, the sample object may be an object of the same type as the target object. Specifically, the specific details of step S301 and step S303 may refer to the specific details of step S201 and step S203, which will not be repeated here.
[0128] In a specific embodiment, the first preset neural network may be a shape representation network to be trained, and the second preset neural network may be an image rendering network to be trained. Specifically, the network structures of the first preset neural network and the second preset neural network may be set in combination with actual applications; optionally, the first preset neural network may be a network including multiple fully connected layers, or a network including convolutional layers, etc.; optionally, the second preset neural network may be a network including multiple fully connected layers, or a network including convolutional layers, etc. The trained shape representation network has the same network structure as the first preset neural network, but different network parameters; the trained image rendering network has the same network structure as the second preset neural network, but different network parameters.
[0129] In a specific embodiment, the sample object shape representation information may be distance information from each sample pixel point to a geometric shape boundary of the sample object.
[0130] In an optional embodiment, if Figure 4 As shown, the training of the first preset neural network and the second preset neural network based on the plurality of sample rendering images and the plurality of sample object images to obtain the shape representation network and the image rendering network may include the following steps:
[0131] In step S401, a second target loss is determined according to a plurality of sample rendering images and a plurality of sample object images;
[0132] In step S403, when the second target loss does not satisfy the second preset condition, the sample object feature, the plurality of sample object difference attribute features, the sample color feature, the first network parameter of the first preset neural network, and the second network parameter of the second preset neural network are updated;
[0133] In step S405, based on the updated sample object features, the updated multiple sample object difference attribute features, the updated sample color features, the updated first preset neural network, and the updated second preset neural network, the pixel coordinates of each sample pixel point, the sample object features, and the multiple sample object difference attribute features are repeatedly input into the first preset neural network to characterize the object shape, so as to obtain sample object shape characterization information, and then a step of determining a second target loss according to the multiple sample rendered images and the multiple sample object images is performed until the second target loss satisfies the second preset condition;
[0134] In step S407, when the second target loss satisfies the second preset condition, the current first preset neural network is used as the shape representation network, and the current second preset neural network is used as the image rendering network.
[0135] In a specific embodiment, according to the plurality of sample rendered images and the plurality of sample object images, determining the second target loss may include determining the loss information between each sample object image and the corresponding sample rendered image based on the first preset loss function, and then adding the loss information of each sample object image pair to obtain the second target loss. Specifically, the second target loss may characterize the difference between the plurality of sample rendered images and the respective corresponding sample object images.
[0136] In a specific embodiment, the first preset loss function may include but is not limited to an L1 norm loss function, a cross entropy loss function, a mean square error loss function, a logical loss function, an exponential loss function, and the like.
[0137] In a specific embodiment, the gradient descent method can be combined to update the sample object features, multiple sample object difference attribute features, sample color features, the first network parameters of the first preset neural network and the second network parameters of the second preset neural network.
[0138] In a specific embodiment, the second target loss satisfies the second preset condition that the second target loss is less than or equal to a specified threshold, or the difference between the corresponding second target losses in two previous and subsequent training processes is less than a certain threshold. In the embodiments of this specification, the specified threshold and the certain threshold can be set in combination with actual training needs.
[0139] In the above embodiment, during the network training process, the pixel coordinates of the pixel points in the multiple sample object images under multiple perspectives, the sample object features and the multiple sample object difference attribute features under multiple perspectives are combined to perform object shape representation learning, and the learned sample object shape representation information is rendered in combination with the image rendering network, and the sample object shape representation information can be combined with the sample rendering image and the sample object image to perform object reconstruction and update, thereby improving the accuracy of the shape representation network for the sample object shape representation. In addition, during the network training process, the sample object features, the multiple sample object difference attribute features and the sample color features are updated while the first network parameters and the second network parameters are updated, so that the first preset neural network can be optimized for the representation ability of the object shape, and the accuracy of the sample object features and the sample color features for the representation of the sample object itself, as well as the accuracy of the multiple sample object difference attribute features for the attribute representation of the sample object under multiple perspectives can be improved, thereby better improving the accuracy of the object shape representation of the trained shape representation network.
[0140] In an optional embodiment, in order to reduce the difficulty of learning object shape representation from sample object images of sample objects, the shape representation network to be trained can be initialized in advance. Accordingly, the above method also includes: a first preset neural network generation step, specifically, Figure 5 As shown, it may include:
[0141] In step S501, the three-dimensional coordinates corresponding to each point on the three-dimensional model of the preset object, the labeled object shape representation information of the preset object and the labeled normal vector corresponding to each point are obtained.
[0142] In step S503, the three-dimensional coordinates are input into the initial neural network to characterize the object shape, and a predicted normal vector corresponding to the preset object and predicted object shape characterization information are obtained.
[0143] In step S505, a third target loss is determined according to the predicted normal vector, the predicted object shape representation information, the labeled object shape representation information and the labeled normal vector.
[0144] In step S507, the initial neural network is trained based on the third target loss to obtain a first preset neural network.
[0145] In a specific embodiment, the three-dimensional model can be generated by a related hardware device (such as a camera device) on the terminal. Specifically, the camera device often generates a three-dimensional model of a preset object in combination with a preset lighting environment. In an optional embodiment, taking a face as an example, multiple three-dimensional face models can be obtained, and average processing is performed based on the multiple three-dimensional face models to obtain a three-dimensional average face model. Optionally, the three-dimensional model of the preset object can be a three-dimensional average object (face) model.
[0146] In a specific embodiment, the shape representation information of the labeled object and the labeled normal vector corresponding to each point on the three-dimensional model can be determined in advance in combination with the three-dimensional model.
[0147] In an optional embodiment, the initial neural network may be a shape representation network to be trained, and correspondingly, the first preset neural network may be a neural network obtained by initializing the initial neural network.
[0148] In an optional embodiment, the above-mentioned determination of the third target loss based on the predicted normal vector, the predicted object shape representation information, the labeled object shape representation information and the labeled normal vector may include: determining the first loss information between the predicted normal vector and the labeled normal vector based on the second preset loss function, and determining the second loss information between the predicted object shape representation information and the labeled object shape representation information based on the third preset loss function; then, adding or weighted adding the first loss information and the second loss information to obtain the third target loss. Specifically, the third target loss can represent the difference between the predicted normal vector and the labeled normal vector, as well as the difference between the predicted object shape representation information and the labeled object shape representation information.
[0149] In a specific embodiment, the second preset loss function and the third preset loss function may include but are not limited to an L1 norm loss function, a cross entropy loss function, a mean square error loss function, a logical loss function, an exponential loss function, etc. The first preset loss function, the second preset loss function, and the third preset loss function may be the same loss function or different loss functions.
[0150] In a specific embodiment, training the initial neural network based on the third target loss to obtain the first preset neural network may include updating the network parameters of the initial neural network based on the gradient descent method when the third target loss satisfies the third preset condition, repeating the above steps S503 and S505 based on the updated initial neural network until the third target loss satisfies the third preset condition, and using the current initial neural network as the first preset neural network when the third target loss satisfies the third preset condition.
[0151] In a specific embodiment, the third target loss satisfies the third preset condition that the third target loss is less than or equal to a specified threshold, or the difference between the corresponding third target losses in two previous and subsequent training processes is less than a certain threshold. In the embodiments of this specification, the specified threshold and the certain threshold can be set in combination with actual training needs.
[0152] In the above embodiment, initializing the shape representation network to be trained in combination with the three-dimensional model of the preset object can improve the accuracy of the first preset neural network in representing the object shape, thereby reducing the difficulty of learning the object shape representation from the sample object image of the sample object and improving the training speed of the shape representation network.
[0153] In step S207, the pixel coordinates, object shape representation information, color features and lighting information of each pixel point are input into the image rendering network for image rendering processing to obtain multiple rendered images under multiple perspectives.
[0154] In a specific embodiment, after combining with the shape representation network to learn the object shape representation information of the target object, the object shape representation information corresponding to the target object image is rendered in combination with the image rendering network, and then the rendered image output by the image rendering network can be combined to achieve self-supervised training of the shape representation network, thereby improving the accuracy of the shape representation network for different objects.
[0155] In step S209, object reconstruction and updating are performed on the object shape representation information based on the multiple rendering images and the multiple target object images to obtain three-dimensional reconstruction information of the target object.
[0156] In an optional embodiment, if Figure 6 As shown, the above-mentioned object reconstruction and updating of the object shape representation information based on multiple rendering images and multiple target object images to obtain the three-dimensional reconstruction information of the target object may include the following steps:
[0157] In step S601, a first target loss is determined according to a plurality of rendering images and a plurality of target object images;
[0158] In step S603, when the first target loss does not satisfy the first preset condition, the object feature, the plurality of object difference attribute features and the color feature are updated;
[0159] In step S605, based on the updated object features, the updated multiple object difference attribute features and the updated color features, the pixel coordinates of each pixel point, the object features and the multiple object difference attribute features are repeatedly input into the shape representation network to perform object shape representation to obtain object shape representation information, until a step of determining a first target loss according to the multiple rendered images and the multiple target object images is performed until the first target loss satisfies a first preset condition;
[0160] In step S607, when the first target loss satisfies the first preset condition, the object shape representation information output by the current shape representation network is used as the three-dimensional reconstruction information.
[0161] In a specific embodiment, the specific details of the above determination of the first target loss can refer to the relevant details of the above determination of the second target loss, which will not be repeated here.
[0162] In a specific embodiment, the first target loss satisfies the first preset condition that the first target loss is less than or equal to a specified threshold, or the difference between the first target losses corresponding to two training processes is less than a certain threshold. In the embodiments of this specification, the specified threshold and the certain threshold can be set in combination with actual training needs.
[0163] In the above embodiment, during the object reconstruction and update process, the first target loss determined by combining multiple rendered images and multiple target object images to update object features, multiple object difference attribute features, and color features can improve the accuracy of the object features and color features in representing the target object itself, as well as the accuracy of the multiple object difference attribute features in representing the attributes of the target object under multiple perspectives, thereby better improving the object shape representation accuracy of the shape representation network.
[0164] In an optional embodiment, based on the three-dimensional reconstruction information of the target object, a texture, special effect animation, etc. of the target object can be generated, which can be specifically configured in combination with actual application requirements.
[0165] It can be seen from the technical solutions provided in the above embodiments of this specification that in the embodiments of this specification, the pixel coordinates of pixel points in multiple target object images under multiple perspectives, object features and multiple object difference attribute features under multiple perspectives are combined to perform object shape representation learning, and the learned object shape representation information is rendered in combination with the image rendering network. The object shape representation information can be combined with the rendered image and the target object image to perform object reconstruction and update, thereby improving the accuracy of the shape representation network's shape representation of the target object, and combining the object shape representation that represents the geometric shape boundary distance information from each pixel point to the target object to perform object reconstruction. This can effectively reduce the storage overhead of the object representation while improving the continuity of the reconstructed three-dimensional reconstruction information, thereby achieving a more comprehensive and accurate object shape representation. While improving the resolution of the object reconstruction information, it can meet the application requirements of multiple resolutions, thereby improving the vividness and resolution of the virtual image subsequently generated based on the three-dimensional reconstruction information.
[0166] In an optional embodiment, if Figure 7 As shown, the above method may also include:
[0167] In step S211, the pixel coordinates of each pixel in the plurality of target object images are determined;
[0168] In step S213, the pixel coordinates are encoded to obtain pixel coordinate features;
[0169] In an optional embodiment, the pixel coordinates may be encoded in combination with a Fourier sine-cosine coordinate encoding algorithm to obtain pixel coordinate features.
[0170] Accordingly, the pixel coordinates, object features and multiple object difference attribute features of each pixel are input into the shape representation network to represent the object shape, and the object shape representation information obtained may include:
[0171] Inputting pixel coordinate features, object features and multiple object difference attribute features into a shape representation network to represent the object shape, thereby obtaining object shape representation information;
[0172] The pixel coordinates, object shape representation information, color features and lighting information of each pixel point are input into the image rendering network for image rendering processing, and multiple rendered images under multiple perspectives are obtained, which may include:
[0173] The pixel coordinate features, object shape representation information, color features and lighting information are input into the image rendering network for image rendering processing to obtain multiple rendered images.
[0174] Optionally, during the training of the shape representation network and the image rendering network, the sample pixel coordinates may also be replaced by sample pixel codes.
[0175] In the above embodiment, the pixel coordinate features after encoding the pixel coordinates are input into the shape representation network for learning, thereby reducing the learning difficulty of the shape representation network for high-frequency information, effectively reducing the learning difficulty of the network, and thereby improving the learning speed of the shape representation network.
[0176] Figure 8 is a block diagram of an object reconstruction device according to an exemplary embodiment. Figure 8 , the device comprises:
[0177] The information acquisition module 810 is configured to acquire a plurality of target object images of the target object captured from multiple perspectives and illumination information of each pixel in the plurality of target object images;
[0178] The feature generation module 820 is configured to generate an object feature of a target object, a color feature of the target object, and a plurality of object difference attribute features corresponding to the target object under multiple viewing angles;
[0179] The first object shape characterization module 830 is configured to input the pixel coordinates, object features and multiple object difference attribute features of each pixel point into the shape characterization network to perform object shape characterization, and obtain object shape characterization information, wherein the object shape characterization information represents the distance information from each pixel point to the geometric shape boundary of the target object;
[0180] The first image rendering processing module 840 is configured to input the pixel coordinates, object shape representation information, color features and lighting information of each pixel point into the image rendering network for image rendering processing to obtain multiple rendered images under multiple perspectives;
[0181] The object reconstruction and updating module 850 is configured to perform object reconstruction and updating on the object shape representation information based on the multiple rendering images and the multiple target object images to obtain three-dimensional reconstruction information of the target object.
[0182] Optionally, the object reconstruction update module 850 includes:
[0183] A first target loss determining unit is configured to determine a first target loss based on a plurality of rendering images and a plurality of target object images;
[0184] A feature updating unit, configured to update the object feature, the plurality of object difference attribute features and the color feature when the first target loss does not satisfy a first preset condition;
[0185] A first iteration unit is configured to execute the steps of repeatedly inputting the pixel coordinates, object features and multiple object difference attribute features of each pixel point into the shape representation network to perform object shape representation based on the updated object features, the updated multiple object difference attribute features and the updated color features, and obtaining object shape representation information, and determining a first target loss according to multiple rendering images and multiple target object images, until the first target loss meets a first preset condition;
[0186] The three-dimensional reconstruction information determination unit is configured to execute, when the first target loss satisfies a first preset condition, using the object shape representation information output by the current shape representation network as the three-dimensional reconstruction information.
[0187] Optionally, the above device further comprises:
[0188] A pixel coordinate determination module is configured to determine the pixel coordinates of each pixel in the plurality of target object images;
[0189] The encoding processing module is configured to perform encoding processing on the pixel coordinates to obtain pixel coordinate features;
[0190] The first object shape characterization module is further configured to input pixel coordinate features, object features, and multiple object difference attribute features into a shape characterization network to perform object shape characterization to obtain object shape characterization information;
[0191] The first image rendering processing module is also configured to execute image rendering processing by inputting pixel coordinate features, object shape representation information, color features and lighting information into an image rendering network to obtain multiple rendered images.
[0192] Optionally, the feature generation module 820 includes:
[0193] An object feature generating unit, configured to randomly generate object features based on first preset dimension information;
[0194] A color feature generating unit is configured to randomly generate a color feature based on the second preset dimension information;
[0195] The object difference attribute feature generating unit is configured to randomly generate a plurality of object difference attribute features based on the third preset dimension information.
[0196] Optionally, the above device further comprises:
[0197] A first sample information acquisition module is configured to acquire a plurality of sample object images of the sample object collected under multiple viewing angles and sample illumination information of each sample pixel in the plurality of sample object images;
[0198] A sample feature generation module is configured to generate a sample object feature of a sample object, a sample color feature of a sample object, and a plurality of sample object difference attribute features corresponding to the sample object under multiple viewing angles;
[0199] A second object shape characterization module is configured to input pixel coordinates of each sample pixel point, sample object features, and multiple sample object difference attribute features into a first preset neural network to perform object shape characterization, and obtain sample object shape characterization information, where the sample object shape characterization information is distance information from each sample pixel point to a geometric shape boundary of the sample object;
[0200] The second image rendering processing module is configured to input the pixel coordinates of each sample pixel point, the sample object shape representation information, the sample color characteristics and the sample illumination information into a second preset neural network for image rendering processing to obtain a plurality of sample rendering images under multiple viewing angles;
[0201] The first network training module is configured to train a first preset neural network and a second preset neural network based on a plurality of sample rendering images and a plurality of sample object images to obtain a shape representation network and an image rendering network.
[0202] Optionally, the first network training module includes:
[0203] A second target loss determining unit is configured to determine a second target loss based on a plurality of sample rendered images and a plurality of sample object images;
[0204] A training data updating unit is configured to update the sample object feature, the plurality of sample object difference attribute features, the sample color feature, the first network parameter of the first preset neural network, and the second network parameter of the second preset neural network when the second target loss does not satisfy the second preset condition;
[0205] The second iteration unit is configured to execute the steps of repeatedly inputting the pixel coordinates of each sample pixel point, the sample object feature and the multiple sample object difference attribute features into the first preset neural network for object shape characterization to obtain sample object shape characterization information based on the updated sample object feature, the updated multiple sample object difference attribute features, the updated sample color feature, the updated first preset neural network and the updated second preset neural network, and determining the second target loss according to the multiple sample rendering images and the multiple sample object images until the second target loss meets the second preset condition;
[0206] The network determination unit is configured to execute, when the second target loss satisfies the second preset condition, using the current first preset neural network as a shape representation network and using the current second preset neural network as an image rendering network.
[0207] Optionally, the above device further comprises:
[0208] The second sample information acquisition module is configured to execute acquisition of the three-dimensional coordinates corresponding to each point on the three-dimensional model of the preset object, the labeled object shape representation information of the preset object, and the labeled normal vector corresponding to each point;
[0209] A third object shape characterization module is configured to execute the input of the three-dimensional coordinates into the initial neural network to perform object shape characterization, and obtain a predicted normal vector corresponding to the preset object and predicted object shape characterization information;
[0210] A third target loss determination module is configured to determine a third target loss based on the predicted normal vector, the predicted object shape representation information, the labeled object shape representation information and the labeled normal vector;
[0211] The second network training module is configured to execute an initial neural network training based on a third target loss to obtain a first preset neural network.
[0212] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0213] Fig. 9 is a block diagram of an electronic device for object reconstruction according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an object reconstruction method is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.
[0214] Fig.10 is a block diagram of an electronic device for object reconstruction according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as shown in FIG. Fig.10 As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an object reconstruction method is implemented.
[0215] Those skilled in the art will understand that Fig. 9 or Fig.10 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0216] In an exemplary embodiment, an electronic device is also provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the object reconstruction method in the embodiment of the present disclosure.
[0217] In an exemplary embodiment, a computer-readable storage medium is also provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the object reconstruction method in the embodiment of the present disclosure.
[0218] In an exemplary embodiment, a computer program product containing instructions is also provided. When the computer program product is run on a computer, the computer is enabled to perform the object reconstruction method in the embodiment of the present disclosure.
[0219] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0220] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0221] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A method for object reconstruction, characterized in that: include: Acquire multiple target object images of the target object captured from multiple viewing angles and illumination information of each pixel in the multiple target object images; Generate an object feature of the target object, a color feature of the target object, and a plurality of object difference attribute features corresponding to the target object under the multiple viewing angles; the plurality of object difference attribute features represent the target attributes of the target object under the multiple viewing angles, and the target attributes are attributes in the attribute information of the target object that may differ due to different viewing angles; Inputting the pixel coordinates of each pixel point, the object feature and the plurality of object difference attribute features into a shape representation network to perform object shape representation, thereby obtaining object shape representation information, wherein the object shape representation information represents the distance information from each pixel point to the geometric shape boundary of the target object; Inputting the pixel coordinates of each pixel point, the object shape representation information, the color feature and the illumination information into an image rendering network for image rendering processing to obtain a plurality of rendered images under the multi-viewing angles; Based on the multiple rendering images and the multiple target object images, the object shape representation information is updated by object reconstruction to obtain three-dimensional reconstruction information of the target object.
2. The object reconstruction method according to claim 1, characterized in that: The performing object reconstruction and updating on the object shape representation information based on the multiple rendered images and the multiple target object images to obtain the three-dimensional reconstruction information of the target object comprises: determining a first target loss based on the plurality of rendered images and the plurality of target object images; When the first target loss does not satisfy a first preset condition, updating the object feature, the plurality of object difference attribute features and the color feature; Based on the updated object feature, the updated multiple object difference attribute features and the updated color feature, repeating the step of inputting the pixel coordinates of each pixel point, the object feature and the multiple object difference attribute features into the shape representation network to perform object shape representation to obtain object shape representation information, to the step of determining the first target loss according to the multiple rendered images and the multiple target object images, until the first target loss meets the first preset condition; When the first target loss satisfies the first preset condition, the object shape representation information output by the current shape representation network is used as the three-dimensional reconstruction information.
3. The object reconstruction method according to claim 1, characterized in that: The method further comprises: Determining pixel coordinates of each pixel in the plurality of target object images; Encoding the pixel coordinates to obtain pixel coordinate features; The step of inputting the pixel coordinates of each pixel point, the object features and the plurality of object difference attribute features into a shape representation network to perform object shape representation to obtain object shape representation information comprises: Inputting the pixel coordinate feature, the object feature and the plurality of object difference attribute features into the shape representation network to perform object shape representation, thereby obtaining the object shape representation information; The step of inputting the pixel coordinates of each pixel point, the object shape representation information, the color feature and the illumination information into an image rendering network for image rendering processing to obtain the plurality of rendered images under the multi-viewing angles comprises: The pixel coordinate features, the object shape representation information, the color features and the illumination information are input into the image rendering network for image rendering processing to obtain the multiple rendered images.
4. The object reconstruction method according to claim 1, characterized in that: The generating of the object feature of the target object, the color feature of the target object and the plurality of object difference attribute features corresponding to the target object under the multiple viewing angles comprises: Based on the first preset dimension information, randomly generate the object feature; Based on the second preset dimension information, randomly generate the color feature; Based on the third preset dimension information, the plurality of object difference attribute features are randomly generated.
5. The object reconstruction method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire a plurality of sample object images of the sample object collected under the multiple viewing angles and sample illumination information of each sample pixel point in the plurality of sample object images; generating a sample object feature of the sample object, a sample color feature of the sample object, and a plurality of sample object difference attribute features corresponding to the sample object under the multiple viewing angles; Inputting the pixel coordinates of each sample pixel point, the sample object feature and the plurality of sample object difference attribute features into a first preset neural network to perform object shape characterization, and obtaining sample object shape characterization information, wherein the sample object shape characterization information is distance information from each sample pixel point to a geometric shape boundary of the sample object; Inputting the pixel coordinates of each sample pixel point, the sample object shape representation information, the sample color feature and the sample illumination information into a second preset neural network for image rendering processing to obtain a plurality of sample rendering images under the multi-viewing angles; Based on the multiple sample rendering images and the multiple sample object images, the first preset neural network and the second preset neural network are trained to obtain the shape representation network and the image rendering network.
6. The object reconstruction method according to claim 5, characterized in that: The training of the first preset neural network and the second preset neural network based on the multiple sample rendering images and the multiple sample object images to obtain the shape representation network and the image rendering network comprises: determining a second target loss based on the plurality of sample rendered images and the plurality of sample object images; When the second target loss does not satisfy a second preset condition, updating the sample object feature, the plurality of sample object difference attribute features, the sample color feature, a first network parameter of the first preset neural network, and a second network parameter of the second preset neural network; Based on the updated sample object feature, the updated multiple sample object difference attribute features, the updated sample color feature, the updated first preset neural network and the updated second preset neural network, repeating the step of inputting the pixel coordinates of each sample pixel point, the sample object feature and the multiple sample object difference attribute features into the first preset neural network for object shape characterization to obtain sample object shape characterization information, to the step of determining the second target loss according to the multiple sample rendered images and the multiple sample object images, until the second target loss meets the second preset condition; When the second target loss satisfies the second preset condition, the current first preset neural network is used as the shape representation network, and the current second preset neural network is used as the image rendering network.
7. The object reconstruction method according to claim 5, characterized in that: The method further comprises: Obtaining the three-dimensional coordinates corresponding to each point on the three-dimensional model of the preset object, the labeled object shape representation information of the preset object, and the labeled normal vector corresponding to each point; Inputting the three-dimensional coordinates into an initial neural network to characterize the object shape, and obtaining a predicted normal vector and predicted object shape characterization information corresponding to the preset object; Determining a third target loss according to the predicted normal vector, the predicted object shape representation information, the labeled object shape representation information, and the labeled normal vector; The initial neural network is trained based on the third target loss to obtain the first preset neural network.
8. An object reconstruction device, characterized in that: include: An information acquisition module is configured to acquire a plurality of target object images of a target object captured from multiple viewing angles and illumination information of each pixel in the plurality of target object images; A feature generation module is configured to generate an object feature of the target object, a color feature of the target object, and a plurality of object difference attribute features corresponding to the target object under the multiple viewing angles; the plurality of object difference attribute features represent target attributes of the target object under the multiple viewing angles, and the target attributes are attributes in the attribute information of the target object that may differ due to different viewing angles; A first object shape characterization module is configured to input the pixel coordinates of each pixel point, the object feature and the plurality of object difference attribute features into a shape characterization network to perform object shape characterization, and obtain object shape characterization information, wherein the object shape characterization information represents the distance information from each pixel point to the geometric shape boundary of the target object; A first image rendering processing module is configured to input the pixel coordinates of each pixel point, the object shape representation information, the color feature and the illumination information into an image rendering network for image rendering processing to obtain a plurality of rendered images under the multi-viewing angles; The object reconstruction update module is configured to perform object reconstruction update on the object shape representation information based on the multiple rendering images and the multiple target object images to obtain three-dimensional reconstruction information of the target object.
9. The object reconstruction device according to claim 8, characterized in that: The object reconstruction update module comprises: A first target loss determining unit is configured to determine a first target loss based on the plurality of rendered images and the plurality of target object images; a feature updating unit, configured to update the object feature, the plurality of object difference attribute features and the color feature if the first target loss does not satisfy a first preset condition; A first iteration unit is configured to execute, based on the updated object feature, the updated multiple object difference attribute features and the updated color feature, repeating the steps of inputting the pixel coordinates of each pixel point, the object feature and the multiple object difference attribute features into a shape representation network to perform object shape representation to obtain object shape representation information, to the step of determining a first target loss according to the multiple rendered images and the multiple target object images, until the first target loss meets the first preset condition; The three-dimensional reconstruction information determination unit is configured to execute, when the first target loss satisfies the first preset condition, using the object shape representation information output by the current shape representation network as the three-dimensional reconstruction information.
10. The object reconstruction device according to claim 8, characterized in that: The device also includes: A pixel coordinate determination module is configured to determine the pixel coordinates of each pixel point in the plurality of target object images; An encoding processing module is configured to perform encoding processing on the pixel coordinates to obtain pixel coordinate features; The first object shape characterization module is further configured to input the pixel coordinate feature, the object feature and the plurality of object difference attribute features into the shape characterization network to perform object shape characterization to obtain the object shape characterization information; The first image rendering processing module is further configured to input the pixel coordinate features, the object shape representation information, the color features and the lighting information into the image rendering network for image rendering processing to obtain the multiple rendered images.
11. The object reconstruction device according to claim 8, characterized in that: The feature generation module comprises: An object feature generating unit, configured to randomly generate the object feature based on the first preset dimension information; A color feature generating unit is configured to randomly generate the color feature based on the second preset dimension information; The object difference attribute feature generating unit is configured to randomly generate the plurality of object difference attribute features based on the third preset dimension information.
12. The object reconstruction device according to any one of claims 8 to 11, characterized in that: The device also includes: A first sample information acquisition module is configured to acquire a plurality of sample object images of the sample object collected under the multiple viewing angles and sample illumination information of each sample pixel in the plurality of sample object images; A sample feature generation module is configured to generate a sample object feature of the sample object, a sample color feature of the sample object, and a plurality of sample object difference attribute features corresponding to the sample object under the multiple viewing angles; A second object shape characterization module is configured to input the pixel coordinates of each sample pixel point, the sample object feature, and the plurality of sample object difference attribute features into a first preset neural network to perform object shape characterization, and obtain sample object shape characterization information, wherein the sample object shape characterization information is distance information from each sample pixel point to a geometric shape boundary of the sample object; A second image rendering processing module is configured to execute image rendering processing by inputting the pixel coordinates of each sample pixel point, the sample object shape representation information, the sample color feature and the sample illumination information into a second preset neural network to obtain a plurality of sample rendering images under the multi-viewing angles; The first network training module is configured to train the first preset neural network and the second preset neural network based on the multiple sample rendering images and the multiple sample object images to obtain the shape representation network and the image rendering network.
13. The object reconstruction device according to claim 12, characterized in that: The first network training module includes: A second target loss determining unit is configured to determine a second target loss based on the plurality of sample rendered images and the plurality of sample object images; a training data updating unit, configured to update the sample object feature, the plurality of sample object difference attribute features, the sample color feature, a first network parameter of the first preset neural network, and a second network parameter of the second preset neural network when the second target loss does not satisfy a second preset condition; The second iteration unit is configured to execute the steps of inputting the pixel coordinates of each sample pixel point, the sample object feature and the multiple sample object difference attribute features into the first preset neural network to characterize the object shape and obtain the sample object shape characterization information, and then determining the second target loss according to the multiple sample rendered images and the multiple sample object images based on the updated sample object feature, the updated multiple sample object difference attribute features, the updated sample color feature, the updated first preset neural network and the updated second preset neural network, until the second target loss meets the second preset condition; The network determination unit is configured to execute, when the second target loss satisfies the second preset condition, using the current first preset neural network as the shape characterization network and using the current second preset neural network as the image rendering network.
14. The object reconstruction device according to claim 12, characterized in that: The device also includes: A second sample information acquisition module is configured to execute acquisition of the three-dimensional coordinates corresponding to each point on the three-dimensional model of the preset object, the labeled object shape representation information of the preset object, and the labeled normal vector corresponding to each point; A third object shape characterization module is configured to execute the input of the three-dimensional coordinates into the initial neural network to perform object shape characterization, and obtain a predicted normal vector and predicted object shape characterization information corresponding to the preset object; A third target loss determination module is configured to determine a third target loss based on the predicted normal vector, the predicted object shape representation information, the labeled object shape representation information and the labeled normal vector; The second network training module is configured to perform training of the initial neural network based on the third target loss to obtain the first preset neural network.
15. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the object reconstruction method according to any one of claims 1 to 7.
16. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the object reconstruction method as claimed in any one of claims 1 to 7.
17. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the object reconstruction method according to any one of claims 1 to 7 is implemented.
