Three-dimensional reconstruction method, color information acquisition method, device and equipment

Abstract

Embodiments of the present disclosure disclose a three-dimensional reconstruction method, a color information acquisition method, an apparatus and a device, wherein the reconstruction method comprises: processing a pixel three-dimensional coordinate set of a pixel point set of a target object surface by using an SDF network to obtain an SDF value set of the pixel point set; processing the pixel three-dimensional coordinate set by using a neural feature network to obtain a neural feature map of the pixel point set; processing, by using a decoding network, based on the SDF value set, the neural feature map and a camera line-of-sight set, to obtain a color information set of the pixel point set; and determining a three-dimensional structure color model of the target object based on a three-dimensional structure model of the target object and the color information set. The embodiments of the present disclosure can realize real-time rendering of the three-dimensional structure model while ensuring a high-quality mesh, and obtain a three-dimensional structure color model of the target object with color information.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of three-dimensional reconstruction, and in particular to a three-dimensional reconstruction method, a color information acquisition method, device and equipment. BACKGROUND

[0002] Neural implicit three-dimensional reconstruction can recover geometric information from pictures taken from multiple angles. The neural implicit reconstruction method in the related art is divided into a signed distance field based method and a neural density field based method. The signed distance field based method can obtain high-quality meshes through post-processing, but cannot render pictures of different angles in real time, because its rendering of color information of these views relies on a large number of computationally expensive neural network inference calculations and volume rendering. The neural density field based method can render the reconstructed object in real time, but its voxel representation has poor universality, or the Mesh quality is poor while the storage and memory occupation are high.

[0003] How to render the reconstructed color in real time on the terminal while ensuring that a high-quality mesh is reconstructed is a problem to be solved. SUMMARY

[0004] Embodiments of the present disclosure provide a three-dimensional reconstruction method, a color information acquisition method, device and equipment to solve the above problems.

[0005] In a first aspect, a three-dimensional reconstruction method is provided, comprising:

[0006] processing, by an SDF network in a color prediction model, a pixel point three-dimensional coordinate set of a pixel point set on a target object surface in a preset three-dimensional coordinate system to obtain an SDF value set of the pixel point set;

[0007] processing, by a neural feature network in the color prediction model, the pixel point three-dimensional coordinate set to obtain a neural feature map of the pixel point set;

[0008] processing, by a decoding network in the color prediction model, based on the SDF value set, the neural feature map and a camera line of sight set respectively facing each pixel point in the pixel point set, to obtain a color information set of the pixel point set;

[0009] determining, based on a three-dimensional structure model of the target object and the color information set, a three-dimensional structure color model of the target object, wherein the three-dimensional structure model is obtained by three-dimensional reconstruction of the target object.

[0010] In an embodiment of the present disclosure, before the SDF network in the color prediction model is used to process a pixel point three-dimensional coordinate set of a pixel point set on a target object surface in a preset three-dimensional coordinate system to obtain an SDF value set of the pixel point set, the method further comprises:

[0011] constructing the SDF network in a virtual space;

[0012] constructing a texture mapping relationship between a texture image including the target object and a three-dimensional coordinate in the preset three-dimensional coordinate system based on the SDF network;

[0013] determining the pixel point three-dimensional coordinate set based on a pixel coordinate set of the pixel point set in the texture image and the texture mapping relationship.

[0014] In an embodiment of the present disclosure, the SDF network is constructed in a virtual space, comprising:

[0015] constructing an initial SDF network in a virtual space;

[0016] performing face reduction processing on the initial SDF network to obtain the SDF network.

[0017] In an embodiment of the present disclosure, the texture mapping relationship between the texture image including the target object and the three-dimensional coordinate in the preset three-dimensional coordinate system is constructed based on the SDF network, comprising:

[0018] performing conformal mapping processing and surface-preserving mapping processing on the SDF network, and constructing the texture mapping relationship based on a result of the conformal mapping processing and the surface-preserving mapping processing.

[0019] In an embodiment of the present disclosure, the pixel point three-dimensional coordinate set is processed by the neural feature network in the color prediction model to obtain a neural feature map of the pixel point set, comprising:

[0020] processing each pixel point three-dimensional coordinate in the pixel point three-dimensional coordinate set by the neural feature network to obtain a four-dimensional neural feature of each pixel point coordinate;

[0021] determining the neural feature map based on the four-dimensional neural features of all pixel points in the pixel point three-dimensional coordinate set.

[0022] In an embodiment of the present disclosure, the color information set of the pixel point set is obtained by the decoding network in the color prediction model based on the SDF value set, the neural feature map and a camera line of sight set of the virtual camera respectively towards each pixel point in the pixel point set, comprising:

[0023] determine a normal vector map of the set of pixels based on the set of SDF values;

[0024] process the normal vector map, the neural feature map, and the set of camera lines of sight using the decoding network to obtain a set of color information of the set of pixels.

[0025] In a second aspect, the present disclosure provides a three-dimensional reconstruction device, comprising:

[0026] a set of SDF value acquisition module configured to process a set of pixel three-dimensional coordinates of a set of pixels on a surface of a target object in a preset three-dimensional coordinate system using an SDF network in a color prediction model to obtain a set of SDF values of the set of pixels;

[0027] a neural feature map acquisition module configured to process the set of pixel three-dimensional coordinates using a neural feature network in the color prediction model to obtain a neural feature map of the set of pixels;

[0028] a set of color information acquisition module configured to process the set of SDF values, the neural feature map, and a set of camera lines of sight of a virtual camera respectively towards each pixel in the set of pixels using a decoding network in the color prediction model to obtain a set of color information of the set of pixels;

[0029] a three-dimensional reconstruction module configured to determine a three-dimensional structure color model of the target object based on a three-dimensional structure model of the target object and the set of color information, wherein the three-dimensional structure model is obtained by three-dimensional reconstruction of the target object.

[0030] In an embodiment of the present disclosure, the three-dimensional reconstruction device further comprises:

[0031] an SDF network construction module configured to construct the SDF network in a virtual space;

[0032] a texture mapping relationship construction module configured to construct a texture mapping relationship between a texture image including the target object and a three-dimensional coordinate in the preset three-dimensional coordinate system based on the SDF network;

[0033] a set of pixel three-dimensional coordinate determination module configured to determine the set of pixel three-dimensional coordinates based on a set of pixel coordinates of the set of pixels in the texture image and the texture mapping relationship.

[0034] In an embodiment of the present disclosure, the SDF network construction module is configured to construct an initial SDF network in a virtual space, and perform face reduction processing on the initial SDF network to obtain the SDF network.

[0035] In an embodiment of the present disclosure, the texture mapping relationship construction module is configured to perform conformal mapping processing and surface-preserving mapping processing on the SDF network, and construct the texture mapping relationship based on results of the conformal mapping processing and the surface-preserving mapping processing.

[0036] In an embodiment of the present disclosure, the neural feature map acquisition module is configured to process each pixel point three-dimensional coordinate in the set of pixel point three-dimensional coordinates by using the neural feature network, to obtain a four-dimensional neural feature of each pixel point coordinate, and determine the neural feature map based on the four-dimensional neural features of all pixel points in the set of pixel point three-dimensional coordinates.

[0037] In an embodiment of the present disclosure, the three-dimensional reconstruction module is configured to determine a normal vector map of the set of pixel points based on the set of SDF values, and process the normal vector map, the neural feature map and the set of camera lines of sight by using the decoding network, to obtain a set of color information of the set of pixel points.

[0038] In a third aspect of the embodiments of the present disclosure, a method for acquiring color information is provided, which includes:

[0039] processing, by using an SDF network in a color prediction model, a pixel point three-dimensional coordinate of a target pixel point on a surface of a target object under a preset three-dimensional coordinate system, to obtain an SDF value of the target pixel point;

[0040] processing, by using a neural feature network in the color prediction model, the pixel point three-dimensional coordinate, to obtain a neural feature of the target pixel point;

[0041] processing, by using a decoding network in the color prediction model, the SDF value, the neural feature and a camera line of sight of a virtual camera towards the target pixel point, to obtain color information of the target pixel point.

[0042] In an embodiment of the present disclosure, before the processing, by using an SDF network in a color prediction model, a pixel point three-dimensional coordinate of a target pixel point on a surface of a target object under a preset three-dimensional coordinate system, to obtain an SDF value of the target pixel point, the method further includes:

[0043] constructing the SDF network in a virtual space;

[0044] constructing, based on the SDF network, a texture mapping relationship between a texture image including the target object and a three-dimensional coordinate under the preset three-dimensional coordinate system;

[0045] determining the pixel point three-dimensional coordinate based on a pixel coordinate of the target pixel point in the texture image and the texture mapping relationship.

[0046] In an embodiment of the present disclosure, the constructing the SDF network in the virtual space comprises:

[0047] constructing an initial SDF network in the virtual space;

[0048] performing a surface reduction process on the initial SDF network to obtain the SDF network.

[0049] In an embodiment of the present disclosure, the constructing, based on the SDF network, a texture mapping relationship between a texture image including the target object and a three-dimensional coordinate in the preset three-dimensional coordinate system comprises:

[0050] performing a conformal mapping process and a surface-preserving mapping process on the SDF network, and constructing the texture mapping relationship based on a result of the conformal mapping process and the surface-preserving mapping process.

[0051] In an embodiment of the present disclosure, the processing, by the decoding network in the color prediction model, based on the SDF value, the neural feature, and a camera line of sight of a virtual camera towards the target pixel point, to obtain the color information of the target pixel point comprises:

[0052] determining a normal vector of the target pixel point based on the SDF value;

[0053] processing, by the decoding network, the normal vector, the neural feature, and the camera line of sight to obtain the color information.

[0054] A fourth aspect of an embodiment of the present disclosure provides a color information acquisition device, comprising:

[0055] an SDF value acquisition unit configured to process, by an SDF network in a color prediction model, a pixel point three-dimensional coordinate of a target pixel point on a surface of a target object in a preset three-dimensional coordinate system to obtain an SDF value of the target pixel point;

[0056] a neural feature acquisition unit configured to process, by a neural feature network in the color prediction model, the pixel point three-dimensional coordinate to obtain a neural feature of the target pixel point;

[0057] a color information acquisition unit configured to process, by a decoding network in the color prediction model, based on the SDF value, the neural feature, and a camera line of sight of a virtual camera towards the target pixel point, to obtain color information of the target pixel point.

[0058] In an embodiment of the present disclosure, the color information acquisition unit further comprises:

[0059] An SDF network construction unit is configured to construct the SDF network in a virtual space.

[0060] A texture mapping relationship construction unit is configured to construct, based on the SDF network, a texture mapping relationship between a texture image of the target object and a three-dimensional coordinate in the preset three-dimensional coordinate system.

[0061] A pixel point three-dimensional coordinate determination unit is configured to determine the pixel point three-dimensional coordinate based on the pixel coordinate of the target pixel point in the texture image and the texture mapping relationship.

[0062] In an embodiment of the present disclosure, the SDF network construction unit is configured to construct an initial SDF network in a virtual space, and perform face reduction processing on the initial SDF network to obtain the SDF network.

[0063] In an embodiment of the present disclosure, the texture mapping relationship construction unit is configured to perform conformal mapping processing and surface-preserving mapping processing on the SDF network, and construct the texture mapping relationship based on a result of the conformal mapping processing and the surface-preserving mapping processing.

[0064] In an embodiment of the present disclosure, the color information acquisition unit is configured to determine a normal vector of the target pixel point based on the SDF value, and perform processing on the normal vector, the neural feature, and the camera line of sight by using the decoding network to obtain the color information.

[0065] In a fifth aspect of the embodiments of the present disclosure, an electronic device is provided, and the electronic device comprises:

[0066] A memory is configured to store a computer program product.

[0067] A processor is configured to execute the computer program product stored in the memory, and when the computer program product is executed, the method in the first aspect is implemented.

[0068] In a sixth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, and the computer readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method in the first aspect is implemented.

[0069] The three-dimensional reconstruction method, the color information acquisition method, the device and the equipment of the embodiment of the present disclosure utilize the SDF network to process the pixel three-dimensional coordinate set of the pixel point set of the target object surface to obtain the SDF value set, utilize the neural feature network to process the pixel three-dimensional coordinate set to obtain the neural feature map, and utilize the lightweight decoding network to process the SDF value set, the neural feature map, and the camera line-of-sight set of each pixel point in the pixel point set to obtain the color information set of the pixel point set, and then according to the three-dimensional structure model of the target object without three-channel color information and the color information set of the pixel point set with three-channel color information, the three-dimensional structure color model of the target object with color information can be obtained. The embodiment of the present disclosure processes the pixel three-dimensional coordinate set, the neural feature map and the normal vector map through the lightweight decoding network, so that the color information set corresponding to the pixel point set on the surface of the target object can be obtained in real time, thereby realizing real-time rendering of the three-dimensional reconstruction model and obtaining the three-dimensional structure color model of the target object with color information.

[0070] The technical solutions of the present disclosure will be further described in detail below by means of the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0071] The accompanying drawings, which form a part of the specification, illustrate the embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0072] The present disclosure can be more clearly understood and appreciated from the following detailed description, taken in conjunction with the accompanying drawings, in which:

[0073] Figure 1 A flowchart of a three-dimensional reconstruction method according to an embodiment of the present disclosure is shown in FIG. 1.

[0074] Figure 2 A structural block diagram of a three-dimensional reconstruction device according to an embodiment of the present disclosure is shown in FIG. 2.

[0075] Figure 3 A flowchart of a color information acquisition method according to an embodiment of the present disclosure is shown in FIG. 3.

[0076] Figure 4 A structural block diagram of a color information acquisition device according to an embodiment of the present disclosure is shown in FIG. 4.

[0077] Figure 5 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown in FIG. 5. DETAILED DESCRIPTION

[0078] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. Note that the relative arrangement, numerical expressions, and numerical values of components and steps set forth in these embodiments are not limiting to the scope of the present disclosure unless otherwise specifically stated.

[0079] Those skilled in the art can understand that the terms "first", "second" and the like in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meaning, nor indicate their logical order.

[0080] It should also be understood that in the embodiments of the present disclosure, "multiple" can mean two or more, and "at least one" can mean one, two or more.

[0081] It should also be understood that for any component, data or structure mentioned in the embodiments of the present disclosure, it can be understood as one or more in general, without explicit limitation or in the context of the opposite indication.

[0082] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in the present disclosure generally represents an "or" relationship between the front and rear associated objects.

[0083] It should also be understood that the description of the embodiments of the present disclosure focuses on the differences between the embodiments, and the same or similar parts can be referred to each other, and for the sake of brevity, will not be repeated.

[0084] The following description of at least one example embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application or uses.

[0085] The techniques, methods and devices known to those of ordinary skill in the relevant art can not be discussed in detail, but in appropriate cases, the techniques, methods and devices should be considered as part of the specification.

[0086] It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0087] The embodiments of the present disclosure can be applied to terminal devices, computer systems, servers and other electronic devices, which can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with terminal devices, computer systems, servers and other electronic devices include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, mainframe computer systems and distributed cloud computing technology environments including any of the above systems, etc.

[0088] Electronic devices such as terminal devices, computer systems, servers, and the like can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like that perform particular tasks or implement particular abstract data types. Computer systems / server can be practiced in distributed cloud computing environments with other computer systems coupled via communication networks. The program modules can be located in local or remote computer system storage media including memory storage devices.

[0089] Figure 1 A flowchart of a three-dimensional reconstruction method in an embodiment of the present disclosure is shown. As shown in the figure, the three-dimensional reconstruction method includes the following steps: Figure 1

[0090] S110: Using a signed distance field (SDF) network in a color prediction model, processing a pixel three-dimensional coordinate set of a pixel point set on a target object surface in a preset three-dimensional coordinate system to obtain an SDF value set of the pixel point set.

[0091] The preset three-dimensional coordinate system can be a virtual three-dimensional space coordinate system established in a virtual space, or a world coordinate system established in a real world, or a camera coordinate system established at a position of a virtual camera.

[0092] When a three-dimensional structure model of a target object is obtained by three-dimensional structure reconstruction of the target object, coordinates of all pixel points on a surface of the target object in a preset three-dimensional coordinate system can be obtained. A set composed of the coordinates of all pixel points on the surface of the target object in the preset three-dimensional coordinate system is referred to as a pixel three-dimensional coordinate set.

[0093] The pixel three-dimensional coordinates can be used as an input of the SDF network in the pre-trained color prediction model, or the encoded pixel three-dimensional coordinates can be used as an input of the SDF network, and the SDF network is used for processing to output an SDF value of each pixel point.

[0094] A set composed of the SDF values corresponding to all pixel points is referred to as an SDF value set.

[0095] S120: Using a neural feature network in a color prediction model, processing the pixel three-dimensional coordinate set to obtain a neural feature map of the pixel point set.

[0096] ​The pixel point three-dimensional coordinates can be used as an input of a neural feature network in the pre-trained color prediction model, or the encoded pixel point three-dimensional coordinates can be used as an input of the neural feature network, and the neural feature network is used for processing to output the neural feature of each pixel point. The neural feature is used for color decoding in a subsequent step.

[0097] A set composed of the neural features corresponding to all the pixel points is referred to as a neural feature map.

[0098] S130: A decoding network in the color prediction model is used for processing based on the SDF value set, the neural feature map and the camera line-of-sight set of the virtual camera towards each pixel point in the pixel point set to obtain a color information set of the pixel point set.

[0099] After the gradient unitization of each SDF value in the SDF value set, a three-channel normal vector corresponding to each SDF value is obtained.

[0100] For each pixel point in the pixel point set, a camera line-of-sight of the virtual camera towards the pixel point can be obtained according to the camera three-dimensional coordinates of the virtual camera in the preset three-dimensional coordinate system and the pixel point three-dimensional coordinates of the pixel point in the preset three-dimensional coordinate system.

[0101] The pixel point three-dimensional coordinates, the normal vector and the camera line-of-sight of the virtual camera towards the pixel point of each pixel point are used as an input of a lightweight decoding network in the color prediction model, and the decoding network is used for processing to output three-channel color information of each pixel point. The three channels can be RGB three channels. The three-channel color of each pixel point in the pixel point set forms the color information set.

[0102] S140: A three-dimensional structure color model of the target object is determined based on the three-dimensional structure model of the target object and the color information set. The three-dimensional structure model is obtained by three-dimensional reconstruction of the target object.

[0103] A three-dimensional structure model of the target object is obtained. The three-dimensional structure model of the target object can be obtained by three-dimensional reconstruction based on multiple images including the target object. The three-dimensional structure model of the target object only includes three-dimensional structure information of the target object and does not include color information of the pixel point set on the surface of the target object.

[0104] The three-dimensional structure model of the target object is rendered in real time based on the color information set, and a three-dimensional structure color model of the target object with color information is rendered in real time.

[0105] In the embodiment, the pixel point three-dimensional coordinate set, the neural feature map and the normal vector map are processed through the lightweight decoding network, so that the color information set corresponding to the pixel point set on the surface of the target object is obtained in real time, the real-time rendering of the three-dimensional structure model is ensured in the case of obtaining a high-quality mesh, and a three-dimensional structure color model of the target object with color information is obtained.

[0106] In one embodiment of the present disclosure, before S110, the following steps are further included:

[0107] S010: constructing an SDF network in a virtual space.

[0108] In one example of the present disclosure, a K*K*K space is divided into N*N*N parts, and then N*N*N three-dimensional coordinates can be obtained. K and N are both integers greater than 0.

[0109] It should be noted that the pixel point three-dimensional coordinate of all the pixel points + the SDF network = the SDF value set, and the SDF value set can also be recorded as the SDF field.

[0110] S020: constructing a texture mapping relationship between a texture image including the target object and a three-dimensional coordinate in a preset three-dimensional coordinate system based on the SDF network.

[0111] Based on the SDF network, a three-dimensional mesh model of the target object can be generated through a preset mesh modeling method. According to the three-dimensional mesh model of the target object, UV mapping calculation is performed, and according to the UV mapping calculation result, the texture mapping relationship between the texture image including the target object and the three-dimensional coordinate in the preset three-dimensional coordinate system can be constructed.

[0112] S030: determining a pixel point three-dimensional coordinate set based on a pixel coordinate set of the pixel point set in the texture image and the texture mapping relationship.

[0113] For the pixel coordinate of each pixel point in the pixel coordinate set in the texture image, the texture mapping relationship is queried, so that the pixel point three-dimensional coordinate of the pixel point in the preset coordinate system is obtained, and then the pixel point three-dimensional coordinate set is obtained.

[0114] In the embodiment, by constructing the SDF network in the virtual space, and then establishing the texture mapping relationship between the texture image including the target object and the three-dimensional coordinate in the preset three-dimensional coordinate system, the three-dimensional space coordinates corresponding to the pixel points in the texture image can be quickly obtained through the texture mapping relationship, which is helpful to quickly obtain the pixel point three-dimensional coordinate set corresponding to the pixel point set on the surface of the target object.

[0115] In one embodiment of the present disclosure, S010 includes:

[0116] S010-1: constructing an initial SDF network in a virtual space.

[0117] In one example of the present disclosure, when a 1*1*1 space is divided into 512*512*512 parts, 512*512*512 3D coordinates can be obtained.

[0118] The initial SDF network can be obtained by using a Marching Cubes (MC) method for three-dimensional reconstruction of the target object.

[0119] S010-2: performing face reduction processing on the initial SDF network to obtain an SDF network.

[0120] Since the initial SDF network contains too many redundant faces, which seriously affects network transmission loading and real-time rendering, an adaptive face reduction method can be used to perform face reduction processing on the initial SDF network to obtain a simplified SDF network. The adaptive face reduction method can use a Quadric Error Metrics (QEM) method. The QEM method can effectively reduce the number of vertices of a complex mesh while preserving the topological features of the original mesh, and the number of reduced vertices can be customized.

[0121] In this embodiment, after constructing the initial SDF network in the virtual space, the simplified SDF network is obtained by performing face reduction processing on the initial SDF network, which helps to reduce the amount of data transmitted when rendering the model in real time and improve the real-time effect of rendering.

[0122] In one embodiment of the present disclosure, S020 includes: performing conformal mapping processing and face-preserving mapping processing on the SDF network, and constructing a texture mapping relationship based on the results of the conformal mapping processing and the face-preserving mapping processing. Through the conformal mapping processing, the angle of the mapped graph can change slightly while keeping the position of the point unchanged. Through the face-preserving mapping processing, the face element can be kept unchanged.

[0123] In this embodiment, by performing conformal mapping processing and face-preserving mapping processing on the SDF network, a texture mapping relationship with a very small change in graph angle and unchanged face element can be obtained, which helps to improve the pixel position accuracy of real-time rendering.

[0124] In one embodiment of the present disclosure, S120 includes:

[0125] S120-1: using a neural feature network to process each pixel point three-dimensional coordinate in the set of pixel point three-dimensional coordinates to obtain a four-dimensional neural feature of each pixel point coordinate. The four-dimensional neural feature includes the features of the pixel point three-dimensional coordinates input to the neural feature network, and also includes the features of the color decoding information of the pixel point three-dimensional coordinates.

[0126] S120-2: Determine the neural feature map based on the four-dimensional neural features of all the pixels in the set of three-dimensional coordinates of the pixels, that is, compose the four-dimensional neural features corresponding to all the pixels in the set of pixels to obtain the neural feature map.

[0127] In this embodiment, the neural feature network can quickly obtain the neural feature map with decoding information, which helps to improve the decoding efficiency and further helps to improve the efficiency of real-time rendering of the target object.

[0128] In one embodiment of the present disclosure, S130 includes:

[0129] S130-1: Determine the normal vector map of the set of pixels based on the set of SDF values.

[0130] Each SDF value in the set of SDF values is gradient unitized to obtain a three-channel normal vector corresponding to each SDF value. The surface normal vectors corresponding to all SDF values form a normal vector map.

[0131] S130-2: Process the normal vector map, the neural feature map, and the set of camera sight lines using the decoding network to obtain a set of color information.

[0132] Each time a pixel is selected from the set of pixels, and then the normal vector, the neural feature, and the camera sight line corresponding to the currently selected pixel are obtained from the normal vector map, the neural feature map, and the set of camera sight lines. Then, the normal vector, the neural feature, and the camera sight line corresponding to the currently selected pixel are input to the lightweight decoding network for processing, and the three-channel color information of the currently selected pixel is output by the decoding network.

[0133] According to the above method, the three-channel color information corresponding to all the pixels in the set of pixels is obtained. The target object can be rendered in real time while obtaining the three-channel color information corresponding to the pixels in the set of pixels.

[0134] In this embodiment, the set of SDF values can be used to obtain the normal vector map corresponding to the set of pixels, and then the decoding network can be used to process the normal vector map, the neural feature map, and the set of camera sight lines to quickly obtain the set of color information, which helps to render the target object in real time.

[0135] Figure 2 The structure block diagram of the three-dimensional reconstruction device in one embodiment of the present disclosure is shown in FIG. 1. Figure 2 As shown in FIG. 1, the three-dimensional reconstruction device includes:

[0136] The SDF value set acquisition module 210 is configured to process a pixel point three-dimensional coordinate set of a pixel point set on a target object surface in a preset three-dimensional coordinate system by using a directed distance field SDF network in the color prediction model, to obtain an SDF value set of the pixel point set.

[0137] The neural feature map acquisition module 220 is configured to process the pixel point three-dimensional coordinate set by using a neural feature network in the color prediction model, to obtain a neural feature map of the pixel point set.

[0138] The color information set acquisition module 230 is configured to process, based on the SDF value set, the neural feature map and a camera line-of-sight set respectively facing each pixel point in the pixel point set by using a decoding network in the color prediction model, to obtain a color information set of the pixel point set.

[0139] The three-dimensional reconstruction module 240 is configured to determine a three-dimensional structure color model of the target object based on a three-dimensional structure model of the target object and the color information set, wherein the three-dimensional structure model is obtained by three-dimensional reconstruction of the target object.

[0140] In an embodiment of the present disclosure, the three-dimensional reconstruction device further comprises:

[0141] The SDF network construction module is configured to construct the SDF network in the virtual space.

[0142] The texture mapping relationship construction module is configured to construct a texture mapping relationship between a texture image of the target object and a three-dimensional coordinate in the preset three-dimensional coordinate system based on the SDF network.

[0143] The pixel point three-dimensional coordinate set determination module is configured to determine the pixel point three-dimensional coordinate set based on a pixel coordinate set of the pixel point set in the texture image and the texture mapping relationship.

[0144] In an embodiment of the present disclosure, the SDF network construction module is configured to construct an initial SDF network in the virtual space, and perform face reduction processing on the initial SDF network to obtain the SDF network.

[0145] In an embodiment of the present disclosure, the texture mapping relationship construction module is configured to perform conformal mapping processing and surface-preserving mapping processing on the SDF network, and construct the texture mapping relationship based on results of the conformal mapping processing and the surface-preserving mapping processing.

[0146] In an embodiment of the present disclosure, the neural feature map acquisition module 220 is configured to process each pixel point three-dimensional coordinate in the pixel point three-dimensional coordinate set by using the neural feature network, to obtain a four-dimensional neural feature of each pixel point coordinate, and determine the neural feature map based on the four-dimensional neural features of all the pixel points in the pixel point three-dimensional coordinate set.

[0147] In one embodiment of the present disclosure, the three-dimensional reconstruction module 240 is configured to determine a normal vector map of the set of pixel points based on the set of SDF values, and process the normal vector map, the neural feature map, and the set of camera view lines using a decoding network to obtain a set of color information of the set of pixel points.

[0148] It should be noted that the specific implementation of the three-dimensional reconstruction device in the embodiments of the present disclosure is similar to the specific implementation of the three-dimensional reconstruction method in the embodiments of the present disclosure. For details, refer to the description in the three-dimensional reconstruction method part. To avoid redundancy, no further description is given.

[0149] Figure 3 A flowchart of a color information acquisition method in one embodiment of the present disclosure is shown in FIG. 3. As shown in FIG. 3, the color information acquisition method includes the following steps: Figure 3

[0150] S310: Using the SDF network in the color prediction model, processing the pixel point three-dimensional coordinates of the target pixel point on the surface of the target object in the preset three-dimensional coordinate system to obtain the SDF value of the target pixel point.

[0151] The preset three-dimensional coordinate system can be a virtual three-dimensional space coordinate system established in a virtual space, or a world coordinate system established in a real world, or a camera coordinate system established at the position of a virtual camera.

[0152] When performing three-dimensional structure reconstruction on the target object to obtain a three-dimensional structure model of the target object, the coordinates of the target pixel point on the surface of the target object in the preset three-dimensional coordinate system can be obtained. The target pixel point can be any pixel point on the surface of the target object.

[0153] The pixel point three-dimensional coordinates can be used as the input of the SDF network in the pre-trained color prediction model, or the encoded pixel point three-dimensional coordinates can be used as the input of the SDF network, which is processed by the SDF network to output the SDF value of the target pixel point.

[0154] S320: Using the neural feature network in the color prediction model, processing the pixel point three-dimensional coordinates to obtain the neural feature of the target pixel point.

[0155] The pixel point three-dimensional coordinates can be used as the input of the neural feature network in the pre-trained color prediction model, or the encoded pixel point three-dimensional coordinates can be used as the input of the neural feature network, which is processed by the neural feature network to output the neural feature of the target pixel point. The neural feature is used for color decoding in the subsequent steps.

[0156] ​S330: Utilize the decoding network in the color prediction model to process based on the SDF value, the neural feature, and the camera line of sight of the virtual camera towards the target pixel point, to obtain the color information of the target pixel point.

[0157] Gradient unitization is performed on the SDF value to obtain a three-channel normal vector corresponding to the SDF value. According to the camera three-dimensional coordinates of the virtual camera in the preset three-dimensional coordinate system and the pixel point three-dimensional coordinates of the target pixel point in the preset three-dimensional coordinate system, the camera line of sight of the virtual camera towards the target pixel point can be obtained.

[0158] The pixel point three-dimensional coordinates of the target pixel point, the normal vector, and the camera line of sight are taken as inputs of a lightweight decoding network in the color prediction model, and the decoding network is used for processing to output three-channel color information of the target pixel point. The three channels can be RGB three channels.

[0159] In this embodiment, the pixel point three-dimensional coordinates, the neural feature, and the normal vector are processed by the lightweight decoding network, so that the color information of any pixel point on the surface of the target object can be quickly obtained, which helps to improve the color rendering efficiency of the target object surface pixel point.

[0160] In one embodiment of the present disclosure, before S310, further comprising:

[0161] S301: Constructing an SDF network in a virtual space.

[0162] In one example of the present disclosure, the K*K*K space is divided into N*N*N parts, and then N*N*N 3d coordinates can be obtained. K and N are both integers greater than 0.

[0163] It should be noted that the pixel point three-dimensional coordinates of the target pixel point + the SDF network = the SDF value of the target pixel point.

[0164] S302: Based on the SDF network, constructing a texture mapping relationship between a texture image including the target object and three-dimensional coordinates in a preset three-dimensional coordinate system.

[0165] Based on the SDF network, a three-dimensional mesh model of the target object can be generated by a preset mesh modeling method. According to the three-dimensional mesh model of the target object, UV mapping calculation is performed, and according to the UV mapping calculation result, a texture mapping relationship between a texture image including the target object and three-dimensional coordinates in a preset three-dimensional coordinate system can be constructed.

[0166] S303: Based on the pixel coordinates of the target pixel point in the texture image and the texture mapping relationship, determining the pixel point three-dimensional coordinates.

[0167] According to the pixel coordinates of the target pixel point in the texture image, the texture mapping relationship is queried, and the three-dimensional coordinates of the target pixel point in the preset coordinate system are obtained.

[0168] In the embodiment, by constructing the SDF network in the virtual space, the texture image including the target object and the three-dimensional coordinates in the preset three-dimensional coordinate system are established. The three-dimensional coordinates in the texture image corresponding to any pixel point can be quickly obtained through the texture mapping relationship. The color information of any pixel point on the surface of the target object can be quickly obtained by processing the three-dimensional coordinates based on the SDF network, the neural feature network and the decoding network, which helps to improve the rendering efficiency of the pixel points on the surface of the target object.

[0169] In one embodiment of the present disclosure, S301 comprises:

[0170] S301-1: Construct an initial SDF network in a virtual space.

[0171] In one example of the present disclosure, a 1*1*1 space is divided into 512*512*512 parts, and then 512*512*512 three-dimensional coordinates can be obtained.

[0172] The initial SDF network obtained by three-dimensional reconstruction of the target object can be obtained by using the Marching Cubes (MC) method.

[0173] S301-2: Perform face reduction processing on the initial SDF network to obtain the SDF network.

[0174] Since the initial SDF network contains too many redundant facets, it seriously affects network transmission loading and real-time rendering. Therefore, an adaptive face reduction method can be used to perform face reduction processing on the initial SDF network to obtain a simplified SDF network. The adaptive face reduction method can use the Quadic Error Metrics (QEM) method. The QEM method can effectively reduce the number of vertices of a complex mesh while preserving the topological features of the original mesh, and the number of vertices after reduction can be customized.

[0175] In the embodiment, after constructing the initial SDF network in the virtual space, the simplified SDF network is obtained by performing face reduction processing on the initial SDF network, which helps to reduce the amount of data transmitted when rendering the model in real time and improve the real-time effect of rendering.

[0176] In an embodiment of the present disclosure, S302 comprises: performing conformal mapping processing and surface-preserving mapping processing on the SDF network, and constructing a texture mapping relationship based on the results of the conformal mapping processing and the surface-preserving mapping processing. The conformal mapping processing can keep the positions of points unchanged while causing a small change in the angles of the mapped graphics. The surface-preserving mapping processing can keep the surface elements unchanged.

[0177] In the present embodiment, the conformal mapping processing and the surface-preserving mapping processing performed on the SDF network can obtain a texture mapping relationship with small changes in graphic angles and unchanged surface elements, which helps to improve the pixel position accuracy of real-time rendering.

[0178] In an embodiment of the present disclosure, S330 comprises:

[0179] S330-1: determining the normal vector of the target pixel point based on the SDF value.

[0180] S330-2: processing the normal vector, the neural feature, and the camera line of sight using the decoding network to obtain color information.

[0181] A target pixel point is selected from the set of pixel points, and the normal vector, the neural feature, and the camera line of sight corresponding to the target pixel point are correspondingly obtained from the normal vector map, the neural feature map, and the set of camera lines of sight. Then, the normal vector, the neural feature, and the camera line of sight corresponding to the target pixel point are input to the lightweight decoding network for processing, and the decoding network outputs the three-channel color information of the target pixel point.

[0182] In the present embodiment, the set of SDF values can be used to obtain the normal vector map corresponding to the set of pixel points, and then the decoding network can be used to process the normal vector, the neural feature, and the camera line of sight corresponding to the target pixel point, so that the color information of the target pixel point can be quickly obtained, which helps to perform real-time color rendering on the target object.

[0183] Figure 4 FIG. 4 is a structural block diagram of a color information acquisition device according to an embodiment of the present disclosure. As shown in FIG. 4, the color information acquisition device comprises: Figure 4

[0184] The SDF value acquisition unit 410 is configured to process the pixel three-dimensional coordinates of the target pixel point on the surface of the target object in a preset three-dimensional coordinate system using the directed distance field SDF network in the color prediction model, to obtain the SDF value of the target pixel point.

[0185] The neural feature acquisition unit 420 is configured to process the pixel three-dimensional coordinates using the neural feature network in the color prediction model, to obtain the neural feature of the target pixel point.

[0186] ​The color information acquisition unit 430 is configured to acquire color information of a target pixel point by using a decoding network in the color prediction model, based on an SDF value, a neural feature, and a camera line of sight of a virtual camera towards the target pixel point.

[0187] In an embodiment of the present disclosure, the color information acquisition device further comprises:

[0188] The SDF network construction unit is configured to construct an SDF network in a virtual space.

[0189] The texture mapping relationship construction unit is configured to construct a texture mapping relationship between a texture image of the target object and a three-dimensional coordinate in a preset three-dimensional coordinate system based on the SDF network.

[0190] The pixel point three-dimensional coordinate determination unit is configured to determine a three-dimensional coordinate of the target pixel point based on a pixel coordinate of the target pixel point in the texture image and the texture mapping relationship.

[0191] In an embodiment of the present disclosure, the SDF network construction unit is configured to construct an initial SDF network in the virtual space, and perform face reduction processing on the initial SDF network to obtain the SDF network.

[0192] In an embodiment of the present disclosure, the texture mapping relationship construction unit is configured to perform conformal mapping processing and surface-preserving mapping processing on the SDF network, and construct the texture mapping relationship based on results of the conformal mapping processing and the surface-preserving mapping processing.

[0193] In an embodiment of the present disclosure, the color information acquisition unit 430 is configured to determine a normal vector of the target pixel point based on the SDF value, and acquire the color information by using the decoding network to process the normal vector, the neural feature, and the camera line of sight.

[0194] It should be noted that the specific implementation of the color information acquisition device in the embodiments of the present disclosure is similar to the specific implementation of the color information acquisition method in the embodiments of the present disclosure. For details, refer to the description of the color information acquisition method. In order to reduce redundancy, no further description is made.

[0195] In addition, the embodiments of the present disclosure also provide an electronic device, comprising:

[0196] The memory is configured to store a computer program.

[0197] The processor is configured to execute the computer program stored in the memory, and when the computer program is executed, the three-dimensional reconstruction method described in any one of the embodiments of the present disclosure is implemented.

[0198] Next, the electronic device according to the embodiments of the present disclosure will be described with reference to Figure 5 As shown in Figure 5As shown, the electronic device includes one or more processors and a memory.

[0199] The processor can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities and can control other components in the electronic device to perform desired functions.

[0200] The memory can store one or more computer program products, which can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), cache memory, and / or the like. The non-volatile memory, for example, can include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program products can be stored on the computer-readable storage media, and the processor can execute the computer program products to implement the three-dimensional reconstruction method of various embodiments of the present disclosure described above and / or other desired functions.

[0201] In one example, the electronic device can further include an input device and an output device, which are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0202] In addition, the input device can further include, for example, a keyboard, a mouse, and / or the like.

[0203] The output device can output various information, including determined distance information, direction information, and / or the like, to the outside. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and / or the like.

[0204] Of course, in order to simplify, Figure 5 Only some of the components related to the present disclosure among the electronic device are shown in FIG. 1, and components such as a bus, an input / output interface, and / or the like are omitted. In addition, the electronic device can further include any other appropriate components according to a specific application.

[0205] In addition to the above-described method and device, embodiments of the present disclosure can be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform steps of the three-dimensional reconstruction method according to various embodiments of the present disclosure described in the above part of the specification.

[0206] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0207] Furthermore, embodiments of the present disclosure can also be a computer readable storage medium, having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform the steps described in the above part of the specification as a three-dimensional reconstruction method according to various embodiments of the present disclosure.

[0208] The computer readable storage medium can be any combination of one or more computer readable medium(s). The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0209] The above describes the basic principles of the present disclosure in combination with specific embodiments, but it should be noted that the advantages, benefits, effects and the like mentioned in the present disclosure are only examples and are not limiting, and these advantages, benefits, effects and the like cannot be considered as the must-have of each embodiment of the present disclosure. In addition, the above specific details are only for the purpose of example and understanding, and the above details do not limit the present disclosure to the must-use specific details.

[0210] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between each embodiment can be referred to each other. For system embodiments, since they basically correspond to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0211] The block diagrams of the devices, apparatuses, equipment, systems involved in the present disclosure are only as illustrative examples and are not intended to require or imply that the connection, arrangement, configuration must be as shown in the block diagrams. As those skilled in the art will recognize, the devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have", and the like are open-ended words, mean "including but not limited to", and can be used interchangeably. The words "or" and "and" as used herein mean the word "and / or", and can be used interchangeably, unless the context clearly indicates otherwise. The word "such as" as used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0212] The methods and apparatuses of the present disclosure can be implemented in numerous ways. For example, the methods and apparatuses of the present disclosure can be implemented using software, hardware, firmware, or any combination of software, hardware, and firmware. The above described order of steps for the methods is for illustration only and the steps of the methods of the present disclosure are not limited to the order described above unless otherwise specifically stated. Furthermore, in some embodiments, the present disclosure can also be implemented as a program recorded in a recording medium, which includes machine readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the methods according to the present disclosure.

[0213] It is also important to note that the devices, equipment and methods of the present disclosure can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalents of the present disclosure.

[0214] The above description of the disclosed aspects is given for illustrative purposes and for the purpose of enabling any person skilled in the art to make and use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0215] The above description has been given for the purpose of illustration and description. Furthermore, this description does not intend to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A three-dimensional reconstruction method, characterized in that, include: The directed distance field (SDF) network in the color prediction model is used to process the set of three-dimensional coordinates of the pixel set on the surface of the target object in a preset three-dimensional coordinate system to obtain the set of SDF values ​​of the pixel set. The color prediction model includes the directed distance field (SDF) network, the neural feature network, and the decoding network. The outputs of the directed distance field (SDF) network and the neural feature network are connected to the input of the decoding network. The neural feature network in the color prediction model is used to process the three-dimensional coordinate set of the pixels to obtain a neural feature map of the pixel set; the neural feature map is a set of neural features corresponding to all pixels. Using the decoding network in the color prediction model, the color information set of the pixel set is obtained by processing the set of SDF values, the neural feature map, and the camera gaze set of the virtual camera towards each pixel in the pixel set. Based on the three-dimensional structural model of the target object and the color information set, a three-dimensional structural color model of the target object is determined, wherein the three-dimensional structural model is obtained by three-dimensional reconstruction of the target object; Using the decoding network in the color prediction model, the color information set of the pixel set is obtained by processing the SDF value set, the neural feature map, and the camera gaze set of the virtual camera towards each pixel in the pixel set, including: Each SDF value in the SDF value set is normalized to obtain the three-channel normal vector corresponding to each SDF value, and the normal vector map corresponding to the SDF value set is determined. For each pixel in the pixel set, the camera line of sight from the virtual camera toward that pixel is obtained based on the camera's three-dimensional coordinates in the preset three-dimensional coordinate system and the pixel's three-dimensional coordinates in the preset three-dimensional coordinate system, and the camera line of sight set corresponding to the pixel set is determined. The decoding network is used to process the set of three-dimensional coordinates of the pixels, the neural feature map, and the normal vector map to obtain the set of color information corresponding to the set of pixels.

2. The method according to claim 1, characterized in that, Before processing the set of three-dimensional coordinates of the pixel set on the surface of the target object in a preset three-dimensional coordinate system using the SDF network in the color prediction model to obtain the set of SDF values ​​of the pixel set, the method further includes: Construct the SDF network within virtual space; Based on the SDF network, a texture mapping relationship is constructed between the texture image of the target object and the three-dimensional coordinates in the preset three-dimensional coordinate system; The three-dimensional coordinate set of the pixels is determined based on the set of pixel coordinates in the texture image and the texture mapping relationship.

3. The method according to claim 2, characterized in that, The construction of the SDF network in virtual space includes: Construct the initial SDF network within virtual space; The initial SDF network is subjected to surface reduction processing to obtain the SDF network.

4. The method according to claim 2, characterized in that, The step of constructing a texture mapping relationship based on the SDF network, including the texture image of the target object and the three-dimensional coordinates in the preset three-dimensional coordinate system, includes: The SDF network is subjected to conformal mapping and surface-preserving mapping, and the texture mapping relationship is constructed based on the results of conformal mapping and surface-preserving mapping.

5. The method according to claim 2, characterized in that, The step of processing the set of three-dimensional coordinates of the pixels using the neural feature network in the color prediction model to obtain the neural feature map of the pixel set includes: Using the neural feature network, the three-dimensional coordinates of each pixel in the set of three-dimensional coordinates are processed to obtain the four-dimensional neural features of each pixel coordinate. The neural feature map is determined based on the four-dimensional neural features of all pixels in the set of three-dimensional coordinates of the pixels.

6. A three-dimensional reconstruction device, characterized in that, include: The SDF value set acquisition module is used to process the three-dimensional coordinate set of the pixel set on the surface of the target object in a preset three-dimensional coordinate system using the directed distance field SDF network in the color prediction model, so as to obtain the SDF value set of the pixel set; the color prediction model includes the directed distance field SDF network, the neural feature network and the decoding network, and the outputs of the directed distance field SDF network and the neural feature network are connected to the input of the decoding network; The neural feature map acquisition module is used to process the three-dimensional coordinate set of pixels using the neural feature network in the color prediction model to obtain the neural feature map of the pixel set; the neural feature map is a set composed of neural features corresponding to all pixels. The color information set acquisition module is used to utilize the decoding network in the color prediction model to process the SDF value set, the neural feature map, and the camera gaze set of the virtual camera towards each pixel in the pixel set to obtain the color information set of the pixel set. A 3D reconstruction module is used to determine the 3D structural color model of the target object based on the 3D structural model of the target object and the color information set, wherein the 3D structural model is obtained by 3D reconstruction of the target object; The color information set acquisition module is specifically used to perform gradient normalization on each SDF value in the SDF value set to obtain the three-channel normal vector corresponding to each SDF value, and to determine the normal vector map corresponding to the SDF value set. For each pixel in the pixel set, the camera's line of sight towards that pixel is obtained based on the camera's 3D coordinates in a preset 3D coordinate system and the pixel's 3D coordinates in the same system, thus determining the camera line of sight set corresponding to the pixel set. The decoding network is then used to process the pixel 3D coordinate set, the neural feature map, and the normal vector map to obtain the color information set corresponding to the pixel set.

7. A method for acquiring color information, characterized in that, include: The directed distance field (SDF) network in the color prediction model is used to process the three-dimensional coordinates of the target pixels on the surface of the target object in a preset three-dimensional coordinate system to obtain the SDF value of the target pixels. The color prediction model includes the directed distance field (SDF) network, the neural feature network, and the decoding network. The outputs of the directed distance field (SDF) network and the neural feature network are connected to the input of the decoding network. The neural feature network in the color prediction model is used to process the three-dimensional coordinates of the pixel to obtain the neural features of the target pixel. The color information of the target pixel is obtained by using the decoding network in the color prediction model, based on the SDF value, the neural features, and the camera's line of sight from the virtual camera toward the target pixel. The process of using the decoding network in the color prediction model to process the color information of the target pixel based on the SDF value, the neural features, and the camera's line of sight towards the target pixel, includes: The SDF value is normalized to obtain the normal vector of the three channels corresponding to the SDF value; For the target pixel, the camera line of sight from the virtual camera toward the target pixel is obtained based on the camera's three-dimensional coordinates in the preset three-dimensional coordinate system and the pixel's three-dimensional coordinates in the preset three-dimensional coordinate system, and the camera line of sight corresponding to the target pixel is determined. The decoding network is used to process the three-dimensional coordinates of the pixel, the neural features, and the normal vector to obtain the color information corresponding to the target pixel.

8. A device for acquiring color information, characterized in that, include: The SDF value acquisition unit is used to process the three-dimensional coordinates of the target pixel points on the surface of the target object in a preset three-dimensional coordinate system using the directed distance field SDF network in the color prediction model, so as to obtain the SDF value of the target pixel point; the color prediction model includes the directed distance field SDF network, the neural feature network and the decoding network, and the outputs of the directed distance field SDF network and the neural feature network are connected to the input of the decoding network; The neural feature acquisition unit is used to process the three-dimensional coordinates of the pixel using the neural feature network in the color prediction model to obtain the neural features of the target pixel. The color information acquisition unit is used to process the color information of the target pixel by utilizing the decoding network in the color prediction model, based on the SDF value, the neural features, and the camera's line of sight from the virtual camera toward the target pixel; The color information acquisition unit is specifically used to normalize the SDF value to obtain the normal vector of the three channels corresponding to the SDF value; for the target pixel, based on the camera three-dimensional coordinates of the virtual camera in the preset three-dimensional coordinate system and the pixel three-dimensional coordinates of the target pixel in the preset three-dimensional coordinate system, the camera line of sight of the virtual camera toward the target pixel is obtained, and the camera line of sight corresponding to the target pixel is determined. The decoding network is used to process the three-dimensional coordinates of the pixel, the neural features, and the normal vector to obtain the color information corresponding to the target pixel.

9. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor is configured to execute a computer program product stored in the memory, wherein, when the computer program product is executed, it implements the method described in any one of claims 1-5.

Images