A three-dimensional image reconstruction system and method based on spatial super-resolution

Abstract

The present invention discloses a 3D image reconstruction system and method based on spatial super-resolution. The system includes: a radiation manifold generation module, a feature super-resolution module, a jump super-resolution module and an image reconstruction module; the radiation manifold generation module is used to map and render input spatial coordinate points to obtain a low-resolution feature map, a depth map and a low-resolution image; the feature super-resolution module is used to modulate and perform feature super-resolution processing based on the low-resolution feature map and the depth map to obtain a high-resolution image; the jump super-resolution module is used to process the low-resolution image to obtain a super-resolution image; the image reconstruction module obtains the final image based on the real image, the high-resolution image and the super-resolution image. The present invention is simple to operate, does not require expensive scanning equipment, uses a spatial feature super-resolution method, greatly reduces the consumption of video memory, and can achieve high-resolution single-view 3D reconstruction under the condition of low computing resource consumption.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a three-dimensional image reconstruction system and method based on spatial super-resolution. Background Art

[0002] The physical world we live in is essentially three-dimensional. 3D scene reconstruction is one of the core tasks in 3D computer vision and has broad application prospects. In the fields of computer graphics and computer vision, 3D reconstruction technology focuses on restoring the shape, structure, and appearance of real objects. With the rapid development of computer technology, researchers have made significant progress in 3D reconstruction. Today, 3D reconstruction technology has been widely used in many fields such as autonomous driving, robotics, game production, and film production, significantly improving the technical level and user experience of these industries.

[0003] However, the existing 3D reconstruction technology still has some limitations and problems, mainly high cost, complex operation and difficulty in popularization; it requires a lot of GPU computing resources; and it is impossible to reconstruct a high-resolution, 3D consistent 3D model. 3D reconstruction technology is mainly divided into two directions: traditional 3D reconstruction technology based on visual geometry and 3D image generation technology based on deep learning. The traditional 3D reconstruction method based on visual geometry has a long research history and the technology is relatively mature, but its shortcomings are also very obvious. This method requires the use of expensive and complex hardware equipment to scan objects, has high hardware requirements, is difficult to popularize, and performs poorly when reconstructing complex scenes. The 3D image generation method based on deep learning can be further divided into: 3D image generation based on a single RGB image, 3D image generation based on multiple views, and 3D image generation based on point cloud data. The advantages of these methods are that they do not require high data and are relatively simple to operate, but they also have their own problems. For example, the results of 3D image generation based on a single RGB image lack details, and important parts are missing due to the lack of 3D data priors; 3D image generation based on multiple views uses voxel methods to extend 2D convolution to 3D, but increasing the resolution will significantly increase the calculation time; 3D image generation based on point cloud data lacks connectivity in the point cloud, resulting in an uneven surface of the reconstructed object. In addition, due to the limitations of computing resources (such as GPUs), these deep learning-based methods have difficulty reconstructing high-resolution objects with strong 3D consistency. Summary of the invention

[0004] The present invention aims to solve the deficiencies of the prior art and provides the following solutions:

[0005] A three-dimensional image reconstruction system based on spatial super-resolution, comprising: a radiation manifold generation module, a feature super-resolution module, a jump super-resolution module and an image reconstruction module;

[0006] The radiation manifold generation module is used to map and render the input space coordinate points to obtain a low-resolution feature map, a depth map and a low-resolution image;

[0007] The feature super-resolution module is used to perform modulation and feature super-resolution processing based on the low-resolution feature map and the depth map to obtain a high-resolution image;

[0008] The jump super-resolution module is used to process the low-resolution image to obtain a super-resolution image;

[0009] The image reconstruction module obtains a final image based on the real image, the high-resolution image and the super-resolution image.

[0010] Preferably, the radiation manifold generation module comprises: a manifold surface prediction unit, a radiation generation unit, a mapping unit and a first manifold rendering unit;

[0011] The manifold surface prediction unit predicts the input space coordinate points using a lightweight multi-layer perceptron to obtain a number of equivalent manifold surfaces of different levels, and then obtains a sampling point coordinate set by calculating the intersection of each observation ray and the equivalent manifold surface;

[0012] The radiation generation unit is used to process each sampling point in the sampling point coordinate set to obtain the color, volume density and intermediate manifold feature map of each sampling point;

[0013] The mapping unit is used to project the intermediate manifold feature map onto a regular plane grid to obtain the low-resolution feature map;

[0014] The first manifold rendering unit is used to render the sampling points to generate the depth map and the low-resolution image.

[0015] Preferably, the feature super-resolution module includes: a depth modulation unit, a feature super-resolution unit, a fusion upsampling unit and a second manifold rendering unit;

[0016] The depth modulation unit is used to modulate the depth map as geometric information into the low-resolution feature map to obtain a modulated feature map;

[0017] The feature super-resolution unit is used to perform up-sampling based on the depth map and the modulated feature map to obtain a high-resolution radiation map;

[0018] The fusion upsampling unit is used to obtain condition information based on the depth map, extract features of the low-resolution feature map, and then fuse the condition information and the features into the modulated feature map to obtain a fused low-resolution feature map;

[0019] The fusion upsampling unit is further used to upsample the fused low-resolution feature map to the high-resolution radiation map to obtain a high-resolution radiation feature;

[0020] The second manifold rendering unit is used to obtain the high-resolution image based on the high-resolution radiation feature.

[0021] Preferably, the jump super-resolution module comprises: a first super-resolution unit, a second super-resolution unit and a pixel-by-pixel integration unit;

[0022] The first super-resolution unit processes the low-resolution image through a convolution layer and a PixelShuffle layer to obtain a first super-resolution image;

[0023] The second super-resolution unit processes the low-resolution image through an auxiliary feature block to obtain a second super-resolution image;

[0024] The pixel-by-pixel integration unit is used to integrate the first super-resolution image and the second super-resolution image to obtain the super-resolution image.

[0025] Preferably, the image reconstruction module comprises: a progressive discriminator unit, a discrimination adversarial optimization unit and a consistency optimization unit;

[0026] The progressive discriminator unit is used to obtain the final image based on the real image and the high-resolution image;

[0027] The discriminative adversarial optimization unit performs adversarial training on the progressive discriminator unit through the real image and the super-resolution image to improve the quality of the final image;

[0028] The consistency optimization unit utilizes a loss function to minimize the distance between the real image and the super-resolution image, and minimizes the mean square error between the super-resolution image and the high-resolution image, so as to ensure the consistency of the final image with the content in the input image.

[0029] The present invention also provides a three-dimensional image reconstruction method based on spatial super-resolution, the reconstruction method is applied to any of the above-mentioned reconstruction systems, and comprises the following steps:

[0030] S1. Map and render the input space coordinate points to obtain a low-resolution feature map, a depth map, and a low-resolution image;

[0031] S2. Modulation and feature super-resolution processing are performed based on the low-resolution feature map and the depth map to obtain a high-resolution image;

[0032] S3. Processing the low-resolution image to obtain a super-resolution image;

[0033] S4. Obtain a final image based on the real image, the high-resolution image and the super-resolution image.

[0034] Preferably, the S1 comprises:

[0035] The input space coordinate points are predicted using a lightweight multi-layer perceptron to obtain a number of iso-manifold surfaces of different levels, and then a sampling point coordinate set is obtained by calculating the intersection of each observation ray with the iso-manifold surface;

[0036] Processing each sampling point in the sampling point coordinate set to obtain a color, volume density and intermediate manifold feature map of each sampling point;

[0037] Projecting the intermediate manifold feature map onto a regular plane grid to obtain the low-resolution feature map;

[0038] The sampling points are rendered to generate the depth map and the low-resolution image.

[0039] Preferably, S2 includes:

[0040] Modulating the depth map as geometric information into the low-resolution feature map to obtain a modulated feature map;

[0041] Perform upsampling based on the depth map and the modulated feature map to obtain a high-resolution radiation map;

[0042] Obtaining condition information based on the depth map, extracting features of the low-resolution feature map, and then fusing the condition information and the features into the modulated feature map to obtain a fused low-resolution feature map;

[0043] Upsampling the fused low-resolution feature map to the high-resolution radiation map to obtain a high-resolution radiation feature;

[0044] The high-resolution image is obtained based on the high-resolution radiation characteristics.

[0045] Preferably, S3 includes:

[0046] Processing the low-resolution image through a convolution layer and a PixelShuffle layer to obtain a first super-resolution image;

[0047] The auxiliary feature block processes the low-resolution image to obtain a second super-resolution image;

[0048] The first super-resolution image and the second super-resolution image are integrated to obtain the super-resolution image.

[0049] Preferably, S4 includes:

[0050] Obtaining the final image based on the real image and the high-resolution image using a progressive discriminator;

[0051] Performing adversarial training on the progressive discriminator through the real image and the super-resolution image to improve the quality of the final image;

[0052] A loss function is used to minimize the distance between the real image and the super-resolution image, and to minimize the mean square error between the super-resolution image and the high-resolution image, so as to ensure the consistency of the content of the final image with that of the input image.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The invention is simple to operate, does not require expensive scanning equipment, uses a spatial feature super-resolution method, greatly reduces the consumption of video memory, and can achieve high-resolution single-view imaging under the condition of low computing resource consumption. Figure 3 dimensional reconstruction, while using RGB image geometry to train the model and directly implement feature super-resolution in space. It also uses the jump super-resolution branch to improve the quality of the generated image, narrowing the image quality gap between 3D image generation and 2D image generation, thereby achieving high-resolution, strong 3D consistency 3D reconstruction of objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0056] Figure 1 A schematic diagram of the system structure of an embodiment of the present invention;

[0057] Figure 2 is a schematic structural diagram of a radiation generating unit according to an embodiment of the present invention;

[0058] Figure 3 Schematic diagram of the structure of a feature super-resolution unit according to an embodiment of the present invention;

[0059] Figure 4 A schematic diagram of the structure of a jump super-resolution module according to an embodiment of the present invention;

[0060] Figure 5 A schematic diagram of a method flow chart of an embodiment of the present invention;

[0061] Figure 6 The overall framework flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] Embodiment 1

[0065] In this embodiment, if Figure 1 As shown, a three-dimensional image reconstruction system based on spatial super-resolution includes: a radiation manifold generation module, a feature super-resolution module, a jump super-resolution module and an image reconstruction module.

[0066] The radiation manifold generation module is used to map and render the input space coordinate points to obtain a low-resolution feature map, a depth map and a low-resolution image. The radiation manifold generation module includes: a manifold surface prediction unit, a radiation generation unit, a mapping unit and a first manifold rendering unit.

[0067] The manifold surface prediction unit predicts the input space coordinate points using a lightweight multi-layer perceptron to obtain a number of different levels of equivalent manifold surfaces, and then obtains a sampling point coordinate set by calculating the intersection of each observation ray and the equivalent manifold surface. In this embodiment, the manifold is a set of equivalent surfaces in a three-dimensional scalar field. In the manifold surface prediction unit, the three-dimensional scalar field uses a lightweight multi-layer perceptron (MLP) as a predictor. ,in, x represents the spatial coordinates of a three-dimensional scalar field, s represents a scalar value for the prediction, Represents a set of real numbers; uses a predictor You can get N different levels l i The equivalent manifold surface S i :

[0068]

[0069] Get the equivalent manifold surface S i Then, by calculating each observation ray r = o + td and iso-manifold surfacesS i The intersection of .

[0070] The radiation generation unit is used to process each sampling point in the sampling point coordinate set to obtain the color, volume density and intermediate manifold feature map of each sampling point. Figure 2 As shown, the radiation generation unit consists of 8 layers of MLP based on FiLM SIREN; this embodiment takes random potential codes from Gaussian normal distribution z , and then input into an MLP network consisting of 3 linear layers to obtain a set of weights w , and then the weight w and the coordinates of the sampling points x i Input into the radiation generation unit to obtain the color (RGB) and volume density of each sampling point α and intermediate manifold feature maps.

[0071] The mapping unit is used to project the intermediate manifold feature map onto a regular plane grid to obtain a low-resolution feature map. In this embodiment, a 2D CNN method is used to perform super-resolution operation on the intermediate manifold feature map: first, a mapping network based on orthogonal projection is used to project the manifold feature map onto a regular plane grid to obtain a low-resolution feature map. F lr The resolution of this feature map is 64×64 and the number of channels is c =4+32, where 4 represents the number of channels for color and volume density ( RGBα ), 32 is the number of channels of the manifold feature map.

[0072] The first manifold rendering unit is used to render the sampling points to generate a depth map and a low-resolution image. In this embodiment, the first manifold rendering unit uses a differentiable volume rendering formula to calculate the final color of each light ray. C ( r ):

[0073]

[0074] in, T represents the transmittance from the near plane to the far plane, σ represents the volume density, c Indicates color, j Indicates the current sampling point i The previous sampling point; then the input image is rendered to obtain a low-resolution image I lr , while generating a set of depth maps during the rendering process Z lr .

[0075] The feature super-resolution module is used to perform modulation and feature super-resolution processing based on the low-resolution feature map and the depth map to obtain a high-resolution image. The feature super-resolution module includes: a depth modulation unit, a feature super-resolution unit, a fusion upsampling unit and a second manifold rendering unit.

[0076] The depth modulation unit is used to modulate the depth map as geometric information into the low-resolution feature map to obtain a modulated feature map. In this embodiment, the low-resolution image is obtained. I lr and depth map Z lr After that, the depth map Z lr After the transformation of three convolutional layers, the values ​​of scale γ and deviation β are predicted, and then these predicted values ​​are used to modulate the low-resolution feature map F lr , and obtain the modulated characteristic map :

[0077]

[0078] in, k Indicates the depth k Low-resolution feature and depth maps, Denotes element-wise product. Through depth modulation, spatial geometric information is fused into low-resolution features.

[0079] The feature super-resolution unit is used to perform up-sampling based on the depth map and the modulated feature map to obtain a high-resolution radiation map. Figure 3 As shown in the figure, it consists of several residual dense modules (RRDB) with depth modulators and sub-pixel convolution layers for upsampling. Specifically, the RRDB block is an image super-resolution network that combines the advantages of dense connections and residual learning to improve the efficiency of feature transfer and reuse, while also alleviating the gradient vanishing problem when training deep networks. Sub-pixel convolution is an upsampling method that can more efficiently utilize model parameters and computing resources. The feature super-resolution unit is based on the depth map Z lr And the characteristic diagram after modulation As input, a high-resolution radiation map is obtained by upsampling with 2DCNN R hr :

[0080]

[0081] in, H hr and Whr represents a high-resolution image, express RGBα The number of channels.

[0082] The fusion upsampling unit is used to obtain conditional information based on the depth map, extract features of the low-resolution feature map, and then fuse the conditional information and features into the modulated feature map to obtain a fused low-resolution feature map; the fusion upsampling unit is also used to upsample the fused low-resolution feature map to a high-resolution radiation map to obtain a high-resolution radiation feature. Z lr Information is input into the geometric information processing network to obtain conditional information , and use convolutional neural networks to extract low-resolution feature maps F lr Features , then cond and feat Input into the RRDB block together and fused into the modulated feature map In the fusion, we get the low-resolution feature map Each RRDB block contains 4 residual dense networks and a deep modulation layer. The input features are calculated by these layers in turn and the output of each layer is connected to obtain new features of the low-resolution feature map. , and finally, use the upsampling factor The PixelShuffle operation will calculate the new features Upsampling to high-resolution radiance map .

[0083] The second manifold rendering unit is used to obtain a high-resolution image based on the high-resolution radiation feature. In this embodiment, the second manifold rendering unit is used to render the final color value of each light ray, thereby obtaining the final high-resolution image. I hr .

[0084] The jump super-resolution module is used to process the low-resolution image to obtain a super-resolution image.

[0085] The jump super-resolution module includes: a first super-resolution unit, a second super-resolution unit and a pixel-by-pixel integration unit. The first super-resolution unit processes the low-resolution image through a convolution layer and a PixelShuffle layer to obtain a first super-resolution image; the second super-resolution unit processes the low-resolution image through an auxiliary feature block to obtain a second super-resolution image; the pixel-by-pixel integration unit is used to integrate the first super-resolution image with the second super-resolution image to obtain a super-resolution image. In this embodiment, Figure 4As shown in the figure, the jump super-resolution module consists of 4 attention auxiliary feature blocks AAFBlk, a convolution layer, a PixelShuffle layer and an addition layer, wherein the first super-resolution unit consists of a convolution layer and a PixelShuffle layer, and the second super-resolution unit consists of a convolution layer, 4 attention auxiliary feature blocks AAFBlk and a PixelShuffle layer. Specifically, the first super-resolution unit processes the low-resolution image through the convolution layer and the PixelShuffle layer to obtain the first super-resolution image. The second super-resolution unit processes the low-resolution image through the convolution layer, auxiliary feature block and PixelShuffle layer to obtain the second super-resolution image Finally, the pixel-by-pixel integration unit uses the addition layer to integrate the first super-resolution image with the second super-resolution image to obtain a super-resolution image. .

[0086] The image reconstruction module obtains the final image based on the real image, the high-resolution image and the super-resolution image. The image reconstruction module includes: a progressive discriminator unit, a discriminative adversarial optimization unit and a consistency optimization unit.

[0087] The progressive discriminator unit is used to obtain the final image based on the real image and the high-resolution image. In this embodiment, the progressive discriminator D is used to obtain the final image, and the learning process and generation quality of the model are improved by gradually increasing the model complexity.

[0088] The discriminative adversarial optimization unit conducts adversarial training on the progressive discriminator unit through real images and super-resolution images to improve the quality of the final image. p z , p ξ , p real Randomly sample potential codes in z , camera pose ξ and real images I , and then use the non-saturated GAN loss function with R1 regularization to train the network and optimize the model:

[0089]

[0090] in, represents the GAN loss function, represents the latent variables sampled from the distribution in the latent space, E represents the expected value operation, Represents the distribution from the latent space p z Latent variables sampled in , represents the parameters sampled from the parameter space distribution of the radiating manifold generator ξ , represents a real sample sampled from the real data distribution , f express f ( x ) = log (1+ expx ), D represents the representation discriminator, G represents the radiating manifold generator, λ represents the weight coefficient of the gradient penalty term, Represents the vector differential operator; by minimizing this loss function through adversarial training, the generated image is close to the real image.

[0091] The consistency optimization unit uses a loss function to minimize the distance between the real image and the super-resolution image, and minimizes the mean square error between the super-resolution image and the high-resolution image, so as to ensure the consistency of the content of the final image with that of the input image. In this embodiment, in order to improve the quality of the generated high-resolution image and maintain the content consistency of the generated high-resolution multi-view, the following two loss functions are used: (1) Super-resolution loss function: Use L 1 loss function to minimize the super-resolution image I sr and real images I The distance between pixels , to optimize the generated high-resolution image I hr (2) Content consistency loss: Apply the mean square error (MSE) loss function to minimize the super-resolution image quality. I sr and high-resolution images I hr Mean square error between pixels , where N represents the number of pixels in the image. This loss function enforces the consistency of the content of the two images, so that the high-resolution image will not generate some objects that are not in the low-resolution image.

[0092] Embodiment 2

[0093] In this embodiment, if Figure 5 , Figure 6 As shown, a three-dimensional image reconstruction method based on spatial super-resolution includes the following steps:

[0094] S1. Map and render the input space coordinate points to obtain a low-resolution feature map, a depth map, and a low-resolution image.

[0095] S1 includes: using a lightweight multi-layer perceptron to predict the input space coordinate points to obtain several equal-value manifold surfaces of different levels, and then obtaining a sampling point coordinate set by calculating the intersection of each observation ray and the equal-value manifold surface; processing each sampling point in the sampling point coordinate set to obtain the color, volume density and intermediate manifold feature map of each sampling point; projecting the intermediate manifold feature map onto a regular plane grid to obtain a low-resolution feature map; rendering the sampling points to generate a depth map and a low-resolution image.

[0096] S2. Modulation and feature super-resolution processing are performed based on the low-resolution feature map and depth map to obtain a high-resolution image.

[0097] S2 includes: modulating the depth map as geometric information into a low-resolution feature map to obtain a modulated feature map; upsampling based on the depth map and the modulated feature map to obtain a high-resolution radiation map; obtaining conditional information based on the depth map, and extracting features of the low-resolution feature map, and then fusing the conditional information and features into the modulated feature map to obtain a fused low-resolution feature map; upsampling the fused low-resolution feature map to a high-resolution radiation map to obtain a high-resolution radiation feature; and obtaining a high-resolution image based on the high-resolution radiation feature.

[0098] S3. Process the low-resolution image to obtain a super-resolution image.

[0099] S3 includes: processing the low-resolution image through a convolution layer and a PixelShuffle layer to obtain a first super-resolution image; processing the low-resolution image through an auxiliary feature block to obtain a second super-resolution image; integrating the first super-resolution image and the second super-resolution image to obtain a super-resolution image.

[0100] S4. The final image is obtained based on the real image, the high-resolution image and the super-resolution image.

[0101] S4 includes: using a progressive discriminator to obtain a final image based on a real image and a high-resolution image; adversarially training the progressive discriminator through the real image and the super-resolution image to improve the quality of the final image; using a loss function to minimize the distance between the real image and the super-resolution image, and to minimize the mean square error between the super-resolution image and the high-resolution image, so as to ensure the consistency of the final image with the content in the input image.

[0102] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A three-dimensional image reconstruction system based on spatial super-resolution, characterized in that: include: Radiation manifold generation module, feature super-resolution module, jump super-resolution module and image reconstruction module; The radiation manifold generation module is used to map and render the input space coordinate points to obtain a low-resolution feature map, a depth map and a low-resolution image; The feature super-resolution module is used to perform modulation and feature super-resolution processing based on the low-resolution feature map and the depth map to obtain a high-resolution image; The jump super-resolution module is used to process the low-resolution image to obtain a super-resolution image; The image reconstruction module obtains a final image based on the real image, the high-resolution image and the super-resolution image; The radiation manifold generation module includes: a manifold surface prediction unit, a radiation generation unit, a mapping unit and a first manifold rendering unit; The manifold surface prediction unit predicts the input space coordinate points using a lightweight multi-layer perceptron to obtain a number of equivalent manifold surfaces of different levels, and then obtains a sampling point coordinate set by calculating the intersection of each observation ray and the equivalent manifold surface; The radiation generation unit is used to process each sampling point in the sampling point coordinate set to obtain the color, volume density and intermediate manifold feature map of each sampling point; The mapping unit is used to project the intermediate manifold feature map onto a regular plane grid to obtain the low-resolution feature map; The first manifold rendering unit is used to render the sampling points to generate the depth map and the low-resolution image; The jump super-resolution module includes: a first super-resolution unit, a second super-resolution unit and a pixel-by-pixel integration unit; The first super-resolution unit processes the low-resolution image through a convolution layer and a PixelShuffle layer to obtain a first super-resolution image; The second super-resolution unit processes the low-resolution image through an auxiliary feature block to obtain a second super-resolution image; The pixel-by-pixel integration unit is used to integrate the first super-resolution image and the second super-resolution image to obtain the super-resolution image.

2. A three-dimensional image reconstruction system based on spatial super-resolution according to claim 1, characterized in that: The feature super-resolution module includes: a depth modulation unit, a feature super-resolution unit, a fusion upsampling unit and a second manifold rendering unit; The depth modulation unit is used to modulate the depth map as geometric information into the low-resolution feature map to obtain a modulated feature map; The feature super-resolution unit is used to perform up-sampling based on the depth map and the modulated feature map to obtain a high-resolution radiation map; The fusion upsampling unit is used to obtain condition information based on the depth map, extract features of the low-resolution feature map, and then fuse the condition information and the features into the modulated feature map to obtain a fused low-resolution feature map; The fusion upsampling unit is further used to upsample the fused low-resolution feature map to the high-resolution radiation map to obtain a high-resolution radiation feature; The second manifold rendering unit is used to obtain the high-resolution image based on the high-resolution radiation feature.

3. The three-dimensional image reconstruction system based on spatial super-resolution according to claim 1, characterized in that: The image reconstruction module includes: a progressive discriminator unit, a discrimination adversarial optimization unit and a consistency optimization unit; The progressive discriminator unit is used to obtain the final image based on the real image and the high-resolution image; The discriminative adversarial optimization unit performs adversarial training on the progressive discriminator unit through the real image and the super-resolution image to improve the quality of the final image; The consistency optimization unit utilizes a loss function to minimize the distance between the real image and the super-resolution image, and minimizes the mean square error between the super-resolution image and the high-resolution image, so as to ensure the consistency of the final image with the content in the input image.

4. A three-dimensional image reconstruction method based on spatial super-resolution, the reconstruction method being applied to the reconstruction system according to any one of claims 1 to 3, characterized in that: The following steps are involved: S1. Map and render the input space coordinate points to obtain a low-resolution feature map, a depth map, and a low-resolution image; S2. Modulation and feature super-resolution processing are performed based on the low-resolution feature map and the depth map to obtain a high-resolution image; S3. Processing the low-resolution image to obtain a super-resolution image; S4. Obtaining a final image based on the real image, the high-resolution image and the super-resolution image; The S1 includes: The input space coordinate points are predicted using a lightweight multi-layer perceptron to obtain a number of iso-manifold surfaces of different levels, and then a sampling point coordinate set is obtained by calculating the intersection of each observation ray with the iso-manifold surface; Processing each sampling point in the sampling point coordinate set to obtain a color, volume density and intermediate manifold feature map of each sampling point; Projecting the intermediate manifold feature map onto a regular plane grid to obtain the low-resolution feature map; Rendering the sampling points to generate the depth map and the low-resolution image; The S3 includes: Processing the low-resolution image through a convolution layer and a PixelShuffle layer to obtain a first super-resolution image; The auxiliary feature block processes the low-resolution image to obtain a second super-resolution image; The first super-resolution image and the second super-resolution image are integrated to obtain the super-resolution image.

5. The three-dimensional image reconstruction method based on spatial super-resolution according to claim 4, characterized in that: The S2 includes: Modulating the depth map as geometric information into the low-resolution feature map to obtain a modulated feature map; Perform upsampling based on the depth map and the modulated feature map to obtain a high-resolution radiation map; Obtaining condition information based on the depth map, extracting features of the low-resolution feature map, and then fusing the condition information and the features into the modulated feature map to obtain a fused low-resolution feature map; Upsampling the fused low-resolution feature map to the high-resolution radiation map to obtain a high-resolution radiation feature; The high-resolution image is obtained based on the high-resolution radiation characteristics.

6. The three-dimensional image reconstruction method based on spatial super-resolution according to claim 4, characterized in that: The S4 includes: Obtaining the final image based on the real image and the high-resolution image using a progressive discriminator; Performing adversarial training on the progressive discriminator through the real image and the super-resolution image to improve the quality of the final image; A loss function is used to minimize the distance between the real image and the super-resolution image, and to minimize the mean square error between the super-resolution image and the high-resolution image, so as to ensure the consistency of the content of the final image with that of the input image.

Images