Three-dimensional reconstruction model training method and device

Abstract

The present disclosure provides a method and apparatus for training a three-dimensional reconstruction model, relating to the fields of artificial intelligence technology, particularly computer vision, image processing, and deep learning technology, and having applications in the metaverse, robotics, autonomous driving, city mapping, and the like. A specific implementation of the method includes: inputting a sample image into a neural radiation field model, outputting a sample neural radiation field representation; performing frequency domain conversion based on the sample neural radiation field representation to generate a sample frequency domain graph; calculating an overall supervisory error loss based on the sample neural radiation field representation and the sample frequency domain graph; and adjusting the parameters of the neural radiation field model based on the overall supervisory error loss to obtain a three-dimensional reconstruction model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the fields of computer vision, image processing and deep learning technology, and can be applied to the metaverse, robots, autonomous driving, city maps, etc. Background Art

[0002] In the three-dimensional reconstruction task, the neural radiation field method uses two-dimensional images and pose information to learn scenes, train neural networks and continuous functions as three-dimensional representations, and has important application value in the fields of metaverse, robotics, autonomous driving, urban mapping, etc. Summary of the Invention

[0003] The embodiments of the present disclosure provide a three-dimensional reconstruction model training method, apparatus, device, storage medium, and program product.

[0004] In the first aspect, an embodiment of the present disclosure proposes a three-dimensional reconstruction model training method, including: inputting a sample image into a neural radiation field model, and outputting a sample neural radiation field representation; performing frequency domain conversion based on the sample neural radiation field representation to generate a sample frequency domain graph; calculating the overall supervision error loss based on the sample neural radiation field representation and the sample frequency domain graph; and adjusting the parameters of the neural radiation field model based on the overall supervision error loss to obtain a three-dimensional reconstruction model.

[0005] In the second aspect, an embodiment of the present disclosure proposes a three-dimensional reconstruction method, including: acquiring a two-dimensional image; inputting the two-dimensional image into a three-dimensional reconstruction model, and outputting a three-dimensional model, wherein the three-dimensional reconstruction model is obtained by training a neural radiation field model using image domain information and frequency domain information as supervision.

[0006] In the third aspect, an embodiment of the present disclosure proposes a three-dimensional reconstruction model training device, including: a reconstruction module, configured to input a sample image into a neural radiation field model and output a sample neural radiation field representation; a conversion module, configured to perform frequency domain conversion based on the sample neural radiation field representation to generate a sample frequency domain graph; a calculation module, configured to calculate the overall supervision error loss based on the sample neural radiation field representation and the sample frequency domain graph; an adjustment module, configured to adjust the parameters of the neural radiation field model based on the overall supervision error loss to obtain a three-dimensional reconstruction model.

[0007] In a fourth aspect, an embodiment of the present disclosure proposes a three-dimensional reconstruction device, comprising: an acquisition module, configured to acquire a two-dimensional image; a reconstruction module, configured to input the two-dimensional image into a three-dimensional reconstruction model and output a three-dimensional model, wherein the three-dimensional reconstruction model is obtained by training a neural radiation field model using image domain information and frequency domain information as supervision.

[0008] In a fifth aspect, an embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the first aspect or the second aspect.

[0009] In a sixth aspect, an embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable a computer to execute the method described in the first aspect or the second aspect.

[0010] In a seventh aspect, an embodiment of the present disclosure proposes a computer program product, including a computer program, which implements the method described in the first aspect or the second aspect when executed by a processor.

[0011] The key or important features of the embodiments of the present disclosure are not intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Other features, objects, and advantages of the present disclosure will become more apparent upon reading the detailed description of the non-limiting embodiments made with reference to the following drawings. The drawings are provided for a better understanding of the present disclosure and do not constitute a limitation of the present disclosure. Among them:

[0013] Figure 1 is a flowchart of an embodiment of a 3D reconstruction model training method according to the present disclosure;

[0014] Figure 2 is a flowchart of another embodiment of a three-dimensional reconstruction model training method according to the present disclosure;

[0015] Figure 3 is an architectural diagram for implementing the three-dimensional reconstruction model training method of the embodiment of the present disclosure;

[0016] Figure 4 is a flow chart of an embodiment of a three-dimensional reconstruction method according to the present disclosure;

[0017] Figure 5 is a structural schematic diagram of an embodiment of a three-dimensional reconstruction model training device according to the present disclosure;

[0018] Figure 6 is a schematic structural diagram of an embodiment of a three-dimensional reconstruction device according to the present disclosure;

[0019] Figure 7 It is a block diagram of an electronic device used to implement the 3D reconstruction model training method or 3D reconstruction method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0021] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0022] Figure 1 A process 100 of an embodiment of a 3D reconstruction model training method according to the present disclosure is shown. The 3D reconstruction model training method includes the following steps:

[0023] Step 101: Input a sample image into a neural radiation field model and output a sample neural radiation field representation.

[0024] In this embodiment, the execution subject of the three-dimensional reconstruction model training method can input the sample image into the neural radiation field model and output the sample neural radiation field representation.

[0025] The 3D reconstruction model training method is typically executed by a server. The server can be either hardware or software. If the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers or as a single server. If the server is software, it can be implemented as multiple software programs or software modules (for example, to provide distributed services) or as a single software program or software module. This is not specifically limited here.

[0026] The sample image can be a two-dimensional image of the sample object taken from multiple viewpoints. Neural Radiance Field is a novel viewpoint synthesis method with an implicit scene representation that can be used to generate high-quality three-dimensional reconstruction models. The sample image is input into the neural radiance field model, which uses the sample image and pose information to learn the scene, train a neural network, and use a continuous function as a three-dimensional representation, i.e., the sample neural radiance field representation.

[0027] Step 102: Perform frequency domain conversion based on the sample neural radiation field representation to generate a sample frequency domain map.

[0028] In this embodiment, the execution entity may perform frequency domain conversion based on the sample neural radiation field representation to generate a sample frequency domain map. For example, the sample neural radiation field representation may be subjected to a fast Fourier transform to generate a sample frequency domain map.

[0029] Step 103: Calculate the overall supervision error loss based on the sample neural radiation field representation and the sample frequency domain map.

[0030] In this embodiment, the execution entity may calculate the overall supervision error loss based on the sample neural radiation field representation and the sample frequency domain map.

[0031] Since the sample neural radiation field representation is image domain information and the sample frequency domain map is frequency domain information, the overall supervision error loss calculated based on the sample neural radiation field representation and the sample frequency domain map takes into account both image domain information and frequency domain information. On the basis of the supervision loss in the image domain, the frequency domain constraint is explicitly added to supplement the statistical information and structural information. In some embodiments, the overall supervision error loss can be calculated by the following steps:

[0032] First, based on the sample neural radiance field representation, the transparency normalization loss is calculated, where the transparency normalization loss is an image domain constraint.

[0033] It should be noted that the transparency normalization loss can also be replaced by other image-domain losses, such as the reconstruction loss or the SSIM (Structural Similarity) loss. For example, the neural radiance field method, which combines explicit and implicit representations, calculates the image-domain loss. This method maps the features of sample points in 3D space to a high-dimensional space through explicit encoding and adds a reconstruction loss supervised by the image domain. Another example is the neural radiance field method for real-time rendering, which calculates the image-domain loss. This method learns anisotropic features from Gaussian primitives in 3D space through snowball rendering and adds a reconstruction loss and SSIM loss supervised by the image domain.

[0034] Then, based on the sample neural radiation field representation and the sample frequency domain map, the frequency domain consistency error loss is calculated, where the frequency domain consistency error loss is a frequency domain constraint.

[0035] Finally, the transparency normalization loss and the frequency domain consistency error loss are weightedly summed to obtain the overall supervision error loss.

[0036] Step 104: Based on the overall supervision error loss, adjust the parameters of the neural radiation field model to obtain a three-dimensional reconstructed model.

[0037] In this embodiment, the above-mentioned execution entity can adjust the parameters of the neural radiation field model based on the overall supervision error loss until the overall supervision error loss is small enough and the neural radiation field model converges, and the neural radiation field model can be used as a three-dimensional reconstruction model.

[0038] The disclosed embodiment provides a method for three-dimensional reconstruction of neural radiation fields based on frequency domain consistency constraints, which can take into account both image domain information and frequency domain information, and supplement statistical information and structural information. Based on the supervision loss in the image domain, this method explicitly adds frequency domain constraints to reduce the gap between the rendering effect and the real scene to complete the reconstruction task. It has important practical value in the fields of metaverse, robots, autonomous driving, city maps, etc. When applied to an autonomous driving simulation platform, it can improve the realism of the reconstructed scene, improve rendering capabilities, reduce simulation costs, and thus enhance user experience.

[0039] Continue to refer Figure 2 , which shows a process 200 of another embodiment of a 3D reconstruction model training method according to the present disclosure. The 3D reconstruction model training method includes the following steps:

[0040] Step 201: Input a sample image into a neural radiation field model, and output a sample neural radiation field representation.

[0041] In this embodiment, the execution subject of the three-dimensional reconstruction model training method can input the sample image into the neural radiation field model and output the sample neural radiation field representation.

[0042] The 3D reconstruction model training method is typically executed by a server. The server can be either hardware or software. If the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers or as a single server. If the server is software, it can be implemented as multiple software programs or software modules (for example, to provide distributed services) or as a single software program or software module. This is not specifically limited here.

[0043] The sample image can be a two-dimensional image of the sample object taken from multiple viewpoints. Neural Radiance Field is a novel viewpoint synthesis method with an implicit scene representation that can be used to generate high-quality three-dimensional reconstruction models. The sample image is input into the neural radiance field model, which uses the sample image and pose information to learn the scene, train a neural network, and use a continuous function as a three-dimensional representation, i.e., the sample neural radiance field representation.

[0044] Step 202 : Perform volume rendering on the sample neural radiation field representation to obtain a set of sample light clusters.

[0045] In this embodiment, the execution entity may perform volume rendering on the sample neural radiation field representation to obtain a set of sample light clusters.

[0046] Volume rendering uses a field of three-dimensional data and a specific imaging method to produce an image. This is significantly different from traditional rasterization. While traditional rasterization models objects by constructing triangular facets, volume rendering uses cubic voxels. There are also fundamental differences in the imaging methods. Conventional rasterization uses shading operations on vertices and faces for texturing or lighting, while volume rendering projects light rays into the interior of an object and scatters them to create the final color.

[0047] For the representation of the sample neural radiation field, color can be obtained through the volume rendering integral formula. To ensure that the loss constraints for rendering and modeling have more information, the loss constraints in the frequency domain and structure can be combined to ensure the rendering and modeling quality of the neural radiation field model.

[0048] The volume rendering integral formula may be as follows:

[0049] ;

[0050] .

[0051] in, is the sample pixel color, is the sample light cluster, and the entire light path is divided into equally spaced intervals, is the transmittance, is the voxel density, is the distance between two adjacent sampling points, is the color constant in the paragraph, is opacity.

[0052] It should be noted that for each sample pixel color obtained by the volume rendering integral formula, there is a corresponding true pixel color. Alternatively, for each sample ray cluster obtained by the volume rendering integral formula, there is a corresponding true ray cluster.

[0053] Step 203 : Select a preset number of sample light clusters from the sample light cluster set and combine them into a sample square.

[0054] In this embodiment, the execution entity may select a preset number of sample light clusters from the sample light cluster set and combine them into a sample square.

[0055] Typically, a training batch size of The sample light clusters are randomly reassembled into a shape with a side length of The preset number is , is a positive integer. The preset quantity is set to This is because the fast Fourier transform requires the number of sampling points to be an integer power of 2 in order to calculate better and faster.

[0056] Step 204: Perform a fast Fourier transform on the sample square to generate a sample frequency domain graph.

[0057] In this embodiment, the execution entity may perform a fast Fourier transform on the sample square to generate a sample frequency domain graph.

[0058] Typically, a differentiable two-dimensional fast Fourier transform is used to convert a sample square into a two-dimensional sample frequency domain map.

[0059] Step 205 : Calculate the transparency normalization loss based on the sample neural radiation field representation.

[0060] In this embodiment, the execution entity may calculate a transparency normalization loss based on the sample neural radiation field representation, wherein the transparency normalization loss is an image domain constraint used to characterize the transparency error between the sample neural radiation field representation and the true neural radiation field representation.

[0061] Step 206 : Calculate the image domain reconstruction error loss based on the sample ray cluster and the true value ray cluster.

[0062] In this embodiment, the execution entity may calculate the image domain reconstruction error loss based on the sample ray cluster and the true value ray cluster.

[0063] Among them, the image domain reconstruction error loss formula can be as follows:

[0064] .

[0065] in, is the sample pixel color of the sampled c points, is the sample pixel color of sampling f points, is the true pixel color.

[0066] In the specific implementation, since the image domain reconstruction error loss is based on batch pixel light supervision of GPU (Graphics Processing Unit) parallel computing, it can be rewritten as follows:

[0067] .

[0068] in, Represents the total light set during training The number of batch rays selected in Light A collection of is the sample ray cluster, For The corresponding true value ray cluster.

[0069] Step 207 : Calculate the frequency domain reconstruction error loss based on the sample frequency domain image and the true value frequency domain image.

[0070] In this embodiment, the execution entity may calculate the frequency domain reconstruction error loss based on the sample frequency domain image and the true value frequency domain image.

[0071] Since the true frequency domain distribution should be consistent with the inferred sample frequency domain distribution, the minimum absolute value error loss is used to constrain the error between the sample frequency domain map and the true value frequency domain map, and the frequency domain reconstruction error loss can be obtained. The true value frequency domain map can be generated by fast Fourier transforming the true value square composed of the true value ray clusters.

[0072] Step 208 : Perform weighted summation on the image domain reconstruction error loss and the frequency domain reconstruction error loss to obtain the frequency domain consistency error loss.

[0073] In this embodiment, the execution entity may perform a weighted summation of the image domain reconstruction error loss and the frequency domain reconstruction error loss to obtain the frequency domain consistency error loss.

[0074] Among them, the frequency domain consistency error loss formula can be as follows:

[0075] .

[0076] Among them, the two-dimensional Fourier transform is expressed as , the frequency consistency loss weight is .

[0077] In some embodiments, the frequency domain consistency error loss can be used to separate high and low frequencies. By introducing the average proportion of regional high and low frequency loss constraints, the selection weights of high and low frequency regions can be dynamically adjusted respectively. The specific steps can be as follows:

[0078] First, calculate the two adjacent iteration intervals and The average frequency domain consistency error loss.

[0079] Afterwards, the average frequency domain consistency error loss is separated into the high frequency region consistency error loss by high and low frequency masks. and low-frequency region consistency error loss .

[0080] Then, the high-frequency region consistency error loss is calculated Consistency error loss with low-frequency region The ratio of high and low frequency loss constraints in the region is obtained. .

[0081] Finally, based on the average proportion of regional high and low frequency loss constraints and a first preset threshold, dynamically adjusting the learning weights of the high-frequency region and the learning weights of the low-frequency region.

[0082] When the average proportion of high and low frequency loss constraints in the region When the gradient is greater than the first preset threshold, it can be considered that the high-frequency information has a greater impact than the low-frequency information. In this case, a higher learning weight is assigned to the high-frequency region, while the learning weight of the low-frequency region is adjusted. This makes the gradient contribution come more from the high frequency region, making the algorithm more robust and more effective.

[0083] In step 209 , a weighted summation of the transparency normalization loss and the frequency domain consistency error loss is performed to obtain the overall supervision error loss.

[0084] In this embodiment, the execution entity may perform a weighted summation of the transparency normalization loss and the frequency domain consistency error loss to obtain an overall supervision error loss.

[0085] In the early stage of neural radiation field model training, learning about low-frequency signals is very simple and easy to learn. In the later stage of training, the frequency domain consistency error loss constraint improves the rendering and reconstruction effects by compensating for the statistical distribution of high-frequency textures. By separating the high and low frequency regions in the frequency domain, it is found that the ratio of loss constraints in high-frequency regions to loss constraints in low-frequency regions increases with training. Moreover, such a fast convergence speed in the early stage of the frequency domain will superimpose too large gradients on other constraints (such as reconstruction error loss, transparency normalization loss, etc.), causing the loss function to be too oscillating and the convergence to be unstable. At the same time, in the later stage of training, due to the convergence of the neural radiation field model, all error loss functions have decreased. According to the weight setting of the original frequency domain consistency error loss, the gradient generated in the high-frequency region will be insufficient. Therefore, it is necessary to dynamically adjust the weight of the frequency domain consistency error loss based on the training of the neural radiation field model. The specific steps can be as follows:

[0086] First, calculate the two adjacent iteration intervals separately and The average error loss and .

[0087] Then, calculate the two adjacent iteration intervals and The average error loss and The ratio of error loss is obtained .

[0088] Finally, based on the error loss ratio and a second preset threshold , dynamically adjust the weight of the frequency domain consistency error loss .

[0089] When the error loss ratio When it approaches 1, it can be considered that the low-frequency region has been learned to a certain extent, the neural radiation field model has roughly converged, and the basic texture and color information has been learned. At this time, the frequency domain consistency error loss will not have a large gradient that affects the convergence of the overall model. At this time, by setting a threshold, frequency domain consistency constraints are added to complete the structure and global statistical information, and further learn the low-frequency region. Among them, the overall supervision error loss formula can be:

[0090] .

[0091] in, is the transparency normalization loss, is the frequency domain consistency error loss, for The weight of .

[0092] Because the 2D image signals obtained by a camera are random subsamplings of the true distribution of objects in 3D space in the real world, their sampling frequency does not conform to the Nyquist sampling theorem. Therefore, the spatial frequency domain distribution of the collection of 2D image signals is equal to the true frequency domain distribution of objects in 3D space superimposed with a certain amount of quantization noise. The frequency domain consistency error loss explicitly adds frequency domain supervision to the 2D layer, providing the missing global statistical and structural information in 3D space. It can be plug-and-play applied to all mainstream neural radiation field models.

[0093] Figure 3 It is an architectural diagram for implementing the three-dimensional reconstruction model training method of the embodiment of the present disclosure.

[0094] like Figure 3 As shown, an image 301 is input into a neural radiation field model 302 to obtain a neural radiation field representation 303. Neural radiation field representation 303 is volume rendered 304 to calculate image domain loss 305 and frequency domain loss 306. High and low frequency separation is performed on frequency domain loss 306 using a high-low frequency separation training module 307. The frequency domain consistency of the high and low frequency separation is explicitly added to the image domain loss 305 to train neural radiation field model 302.

[0095] Further references Figure 4 , which shows a process 400 of an embodiment of a 3D reconstruction method according to the present disclosure. The 3D reconstruction method includes the following steps:

[0096] Step 401: Acquire a two-dimensional image.

[0097] In this embodiment, the execution subject of the 3D reconstruction method may obtain a 2D image, wherein the 2D image may be an image of the object captured from multiple viewing angles of the object.

[0098] The 3D reconstruction method is typically executed by a server. The server can be either hardware or software. If the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers or as a single server. If the server is software, it can be implemented as multiple software programs or software modules (for example, to provide distributed services) or as a single software program or software module. This is not specifically limited here.

[0099] Step 402: Input the two-dimensional image into the three-dimensional reconstruction model and output the three-dimensional model.

[0100] In this embodiment, the execution entity may input a two-dimensional image into a three-dimensional reconstruction model and output a three-dimensional model.

[0101] The three-dimensional reconstruction model can be obtained by training the neural radiation field model using both image domain information and frequency domain information as supervision. The training process can be as follows:

[0102] First, the sample image is input into the neural radiation field model, and the sample neural radiation field representation is output.

[0103] The sample image can be a two-dimensional image of the sample object taken from multiple viewpoints. Neural Radiance Field is a novel viewpoint synthesis method with an implicit scene representation that can be used to generate high-quality three-dimensional reconstruction models. The sample image is input into the neural radiance field model, which uses the sample image and pose information to learn the scene, train a neural network, and use a continuous function as a three-dimensional representation, i.e., the sample neural radiance field representation.

[0104] Afterwards, frequency domain conversion is performed based on the sample neural radiation field representation to generate a sample frequency domain map.

[0105] For example, a fast Fourier transform is performed on the sample neural radiation field representation to generate a sample frequency domain map.

[0106] Then, the overall supervision error loss is calculated based on the sample neural radiance field representation and the sample frequency domain map.

[0107] Since the sample neural radiance field representation is image-domain information and the sample frequency domain map is frequency-domain information, the overall supervised error loss calculated based on the sample neural radiance field representation and the sample frequency domain map takes into account both image-domain information and frequency-domain information. On top of the supervised loss in the image domain, the frequency domain constraint is explicitly added to supplement the statistical and structural information.

[0108] Finally, based on the overall supervised error loss, the parameters of the neural radiation field model are adjusted to obtain the 3D reconstructed model.

[0109] Based on the overall supervision error loss, the parameters of the neural radiation field model are adjusted until the overall supervision error loss is small enough and the neural radiation field model converges. The neural radiation field model can then be used as a three-dimensional reconstruction model.

[0110] The disclosed embodiment provides a method for three-dimensional reconstruction of neural radiation fields based on frequency domain consistency constraints, which can take into account both image domain information and frequency domain information, and supplement statistical information and structural information. This method explicitly adds frequency domain constraints to reduce the gap between the rendering effect and the real scene to complete the reconstruction task. It has important practical value in the fields of metaverse, robots, autonomous driving, city maps, etc. When applied to an autonomous driving simulation platform, it can improve the realism of the reconstructed scene, improve rendering capabilities, reduce simulation costs, and thus enhance user experience.

[0111] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a 3D reconstruction model training device. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0112] like Figure 5 As shown, the 3D reconstruction model training device 500 of this embodiment may include: a reconstruction module 501, a conversion module 502, a calculation module 503, and an adjustment module 504. The reconstruction module 501 is configured to input a sample image into a neural radiation field model and output a sample neural radiation field representation; the conversion module 502 is configured to perform frequency domain conversion based on the sample neural radiation field representation to generate a sample frequency domain map; the calculation module 503 is configured to calculate the overall supervision error loss based on the sample neural radiation field representation and the sample frequency domain map; and the adjustment module 504 is configured to adjust the parameters of the neural radiation field model based on the overall supervision error loss to obtain a 3D reconstruction model.

[0113] In this embodiment, the specific processing of the reconstruction module 501, the conversion module 502, the calculation module 503 and the adjustment module 504 and the technical effects thereof can be referred to in the respective Figure 1 The relevant descriptions of steps 101-104 in the corresponding embodiment are not repeated here.

[0114] In some optional implementations of this embodiment, the calculation module 503 includes: a first calculation submodule, configured to calculate the transparency normalization loss based on the sample neural radiation field representation; a second calculation submodule, configured to calculate the frequency domain consistency error loss based on the sample neural radiation field representation and the sample frequency domain map; a summation submodule, configured to perform weighted summation of the transparency normalization loss and the frequency domain consistency error loss to obtain the overall supervision error loss.

[0115] In some optional implementations of this embodiment, the conversion module 502 is further configured to: perform volume rendering on the sample neural radiation field representation to obtain a sample light cluster set; select a preset number of sample light clusters from the sample light cluster set and combine them into a sample square; perform fast Fourier transform on the sample square to generate a sample frequency domain map.

[0116] In some optional implementations of this embodiment, the second calculation submodule includes: a first calculation unit, configured to calculate the image domain reconstruction error loss based on the sample light cluster and the true value light cluster; a second calculation unit, configured to calculate the frequency domain reconstruction error loss based on the sample frequency domain map and the true value frequency domain map, wherein the true value frequency domain map is generated by performing a fast Fourier transform on the true value square composed of the true value light clusters; a first summation unit, configured to perform a weighted summation of the image domain reconstruction error loss and the frequency domain reconstruction error loss to obtain the frequency domain consistency error loss.

[0117] In some optional implementations of this embodiment, the second calculation submodule also includes: a third calculation unit, configured to calculate the average frequency domain consistency error loss of two adjacent iterative intervals; a separation unit, configured to separate the average frequency domain consistency error loss into high-frequency area consistency error loss and low-frequency area consistency error loss through high-frequency and low-frequency masks; a fourth calculation unit, configured to calculate the ratio of the high-frequency area consistency error loss to the low-frequency area consistency error loss to obtain the average proportion of regional high and low frequency loss constraints; a first adjustment unit, configured to dynamically adjust the learning weight of the high-frequency area and the learning weight of the low-frequency area based on the average proportion of regional high and low frequency loss constraints and a first preset threshold.

[0118] In some optional implementations of this embodiment, the second calculation unit is further configured to: use a minimum absolute value error loss to constrain the error between the sample frequency domain image and the true value frequency domain image to obtain a frequency domain reconstruction error loss.

[0119] In some optional implementations of this embodiment, the summation submodule includes: a second adjustment unit, configured to dynamically adjust the weight of the frequency domain consistency error loss based on the training status of the neural radiation field model; a second summation unit, configured to use the weight of the frequency domain consistency error loss to weightedly sum the transparency normalization loss and the frequency domain consistency error loss to obtain the overall supervision error loss.

[0120] In some optional implementations of this embodiment, the second adjustment unit is further configured to: respectively calculate the average error loss of two adjacent iterative intervals; calculate the ratio of the average error loss of two adjacent iterative intervals to obtain the error loss ratio; based on the error loss ratio and the second preset threshold, dynamically adjust the weight of the frequency domain consistency error loss.

[0121] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a three-dimensional reconstruction device. Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0122] like Figure 6 As shown, the 3D reconstruction apparatus 600 of this embodiment may include: an acquisition module 601 and a reconstruction module 602. The acquisition module 601 is configured to acquire a 2D image; the reconstruction module 602 is configured to input the 2D image into a 3D reconstruction model and output a 3D model. The 3D reconstruction model is obtained by training a neural radiation field model using both image domain information and frequency domain information as supervision.

[0123] In this embodiment, the 3D reconstruction device 600 includes the specific processing of the acquisition module 601 and the reconstruction module 602 and the technical effects thereof. Figure 4 The relevant descriptions of steps 401-402 in the corresponding embodiment are not repeated here.

[0124] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0125] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0126] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0127] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. Computing unit 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0128] Various components in device 700 are connected to I / O interface 705, including an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, optical disk, etc.; and a communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0129] The computing unit 701 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the 3D reconstruction model training method. For example, in some embodiments, the 3D reconstruction model training method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the 3D reconstruction model training method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the 3D reconstruction model training method by any other suitable means (e.g., via firmware).

[0130] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0132] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0134] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0135] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0136] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions provided by this disclosure can be achieved. This is not limited herein.

[0137] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A 3D reconstruction model training method, comprising: Input the sample image into the neural radiation field model and output the sample neural radiation field representation; Performing frequency domain conversion based on the sample neural radiation field representation to generate a sample frequency domain map; Calculating an overall supervision error loss based on the sample neural radiation field representation and the sample frequency domain map; Based on the overall supervision error loss, adjusting the parameters of the neural radiation field model to obtain a three-dimensional reconstructed model; The calculating of the overall supervision error loss based on the sample neural radiation field representation and the sample frequency domain map includes: Calculating a transparency normalization loss based on the sample neural radiation field representation; Calculating frequency domain consistency error loss based on the sample neural radiation field representation and the sample frequency domain map; Performing a weighted summation on the transparency normalization loss and the frequency domain consistency error loss to obtain the overall supervision error loss; The performing frequency domain conversion based on the sample neural radiation field representation to generate a sample frequency domain graph includes: Performing volume rendering on the sample neural radiation field representation to obtain a set of sample light clusters; Selecting a preset number of sample light clusters from the set of sample light clusters and combining them into a sample square; Performing a fast Fourier transform on the sample square to generate the sample frequency domain image.

2. The method according to claim 1, wherein The calculating the frequency domain consistency error loss based on the sample neural radiation field representation and the sample frequency domain map includes: Calculating image domain reconstruction error loss based on the sample light cluster and the true light cluster; Calculating a frequency domain reconstruction error loss based on the sample frequency domain graph and the true value frequency domain graph, wherein the true value frequency domain graph is generated by performing a fast Fourier transform on a true value square formed by combining the true value light clusters; A weighted sum is performed on the image domain reconstruction error loss and the frequency domain reconstruction error loss to obtain the frequency domain consistency error loss.

3. The method according to claim 2, wherein: The calculating the frequency domain consistency error loss based on the sample neural radiation field representation and the sample frequency domain map further includes: Calculate the average frequency domain consistency error loss between two adjacent iteration intervals; Separating the average frequency domain consistency error loss into high-frequency region consistency error loss and low-frequency region consistency error loss through high- and low-frequency masks; Calculating the ratio of the high-frequency region consistency error loss to the low-frequency region consistency error loss to obtain an average proportion of regional high- and low-frequency loss constraints; Based on the average proportion of the high-frequency and low-frequency loss constraints in the region and a first preset threshold, the learning weight of the high-frequency region and the learning weight of the low-frequency region are dynamically adjusted.

4. The method according to claim 2, wherein: The calculating the frequency domain reconstruction error loss based on the sample frequency domain graph and the true value frequency domain graph includes: The error between the sample frequency domain image and the true value frequency domain image is constrained by using a minimum absolute value error loss to obtain the frequency domain reconstruction error loss.

5. The method according to claim 2, wherein: The weighted summation of the transparency normalization loss and the frequency domain consistency error loss to obtain the overall supervision error loss includes: Dynamically adjusting the weight of the frequency domain consistency error loss based on the training status of the neural radiation field model; The weight of the frequency domain consistency error loss is used to perform weighted summation on the transparency normalization loss and the frequency domain consistency error loss to obtain the overall supervision error loss.

6. The method according to claim 5, wherein: The dynamically adjusting the weight of the frequency domain consistency error loss based on the training condition of the neural radiation field model includes: Calculate the average error loss of two adjacent iteration intervals respectively; Calculating the ratio of the average error losses of the two adjacent iteration intervals to obtain the error loss ratio; The weight of the frequency domain consistency error loss is dynamically adjusted based on the error loss proportion and a second preset threshold.

7. A three-dimensional reconstruction method, comprising: Acquire a two-dimensional image; The two-dimensional image is input into a three-dimensional reconstruction model, and a three-dimensional model is output, wherein the three-dimensional reconstruction model is trained using the method according to any one of claims 1 to 6.

8. A 3D reconstruction model training device, comprising: a reconstruction module configured to input a sample image into a neural radiation field model and output a sample neural radiation field representation; a conversion module configured to perform frequency domain conversion based on the sample neural radiation field representation to generate a sample frequency domain map; a calculation module configured to calculate an overall supervision error loss based on the sample neural radiation field representation and the sample frequency domain map; an adjustment module, configured to adjust parameters of the neural radiation field model based on the overall supervision error loss to obtain a three-dimensional reconstructed model; The calculation module includes: a first calculation submodule, configured to calculate a transparency normalization loss based on the sample neural radiation field representation; A second calculation submodule is configured to calculate a frequency domain consistency error loss based on the sample neural radiation field representation and the sample frequency domain map; a summation submodule, configured to perform a weighted summation on the transparency normalization loss and the frequency domain consistency error loss to obtain the overall supervision error loss; The conversion module is further configured to: Performing volume rendering on the sample neural radiation field representation to obtain a set of sample light clusters; Selecting a preset number of sample light clusters from the set of sample light clusters and combining them into a sample square; Performing a fast Fourier transform on the sample square to generate the sample frequency domain image.

9. The device according to claim 8, wherein The second calculation submodule includes: A first calculation unit is configured to calculate an image domain reconstruction error loss based on the sample light cluster and the true light cluster; a second calculation unit configured to calculate a frequency domain reconstruction error loss based on the sample frequency domain map and a true value frequency domain map, wherein the true value frequency domain map is generated by performing a fast Fourier transform on a true value square formed by combining the true value light clusters; The first summing unit is configured to perform a weighted summation on the image domain reconstruction error loss and the frequency domain reconstruction error loss to obtain the frequency domain consistency error loss.

10. The device according to claim 9, wherein The second calculation submodule further includes: The third calculation unit is configured to calculate the average frequency domain consistency error loss of two adjacent iteration intervals; a separation unit configured to separate the average frequency domain consistency error loss into a high frequency region consistency error loss and a low frequency region consistency error loss through high and low frequency masks; a fourth calculation unit configured to calculate a ratio of the high-frequency region consistency error loss to the low-frequency region consistency error loss to obtain an average proportion of regional high- and low-frequency loss constraints; The first adjustment unit is configured to dynamically adjust the learning weight of the high-frequency region and the learning weight of the low-frequency region based on the average proportion of the high- and low-frequency loss constraints in the region and a first preset threshold.

11. The device according to claim 9, wherein The second computing unit is further configured to: The error between the sample frequency domain image and the true value frequency domain image is constrained by using a minimum absolute value error loss to obtain the frequency domain reconstruction error loss.

12. The device according to claim 9, wherein The summation submodule includes: A second adjustment unit is configured to dynamically adjust the weight of the frequency domain consistency error loss based on the training status of the neural radiation field model; The second summing unit is configured to use the weight of the frequency domain consistency error loss to perform weighted summation on the transparency normalization loss and the frequency domain consistency error loss to obtain the overall supervision error loss.

13. The device according to claim 12, wherein The second adjustment unit is further configured to: Calculate the average error loss of two adjacent iteration intervals respectively; Calculating the ratio of the average error losses of the two adjacent iteration intervals to obtain the error loss ratio; The weight of the frequency domain consistency error loss is dynamically adjusted based on the error loss proportion and a second preset threshold.

14. A three-dimensional reconstruction device comprising: an acquisition module configured to acquire a two-dimensional image; The reconstruction module is configured to input the two-dimensional image into a three-dimensional reconstruction model and output a three-dimensional model, wherein the three-dimensional reconstruction model is trained using the device according to any one of claims 8 to 13.

15. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6 or the method of claim 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method of any one of claims 1 to 6 or the method of claim 7.

17. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 6 or the method according to claim 7.

Images