Information processing device, information processing method, and computer-readable non-transitory storage medium

Abstract

This information processing device is provided with a learning unit and a correcting unit. The learning unit learns radiance fields on the basis of multi-viewpoint images including a plurality of base viewpoint images. The correcting unit corrects the learned radiance fields on the basis of a base viewpoint, which is a spatial correction reference, the base viewpoint images, serving as ground truth (GT) images corresponding to the base viewpoint, and discrete three-dimensional sampling references.

Description

Information processing device, information processing method, and computer-readable non-transitory storage medium

[0001] The present disclosure relates to an information processing device, an information processing method, and a computer-readable non-transitory storage medium.

[0002] Conventionally, an image synthesis technology known as Novel View Synthesis (NVS) is known, which captures a three-dimensional space based on multi-viewpoint ground truth (GT) images from each real viewpoint, and synthesizes and creates an image from a new viewpoint based on the learning results of the radiance field.

[0003] In this NVS, highly accurate reconstruction of the radiance field is important for improving the sense of immersion and realism in XR (Cross Reality) and virtual production.

[0004] As a method for reconstructing the radiance field in three-dimensional space with high accuracy, a method using a neural network called NeRF (Neural Radiance Fields) has been proposed in recent years (see, for example, Non-Patent Document 1).

[0005] NeRF expresses colors that depend on the density and viewing direction of a three-dimensional space as a multilayer perceptron (MLP), and trains the MLP so that the brightness values ​​obtained by volume rendering approach the brightness values ​​of the captured GT image. However, NeRF has issues such as difficulty in real-time rendering for new input viewing directions, a tendency for high-frequency components of texture to be lost, and the need for long learning times.

[0006] To address this issue, a method called NeX (NeXt-level view-dependent effects) has been proposed, which arranges multi-layer images called MPI (Multiplane Images) (see, for example, Non-Patent Document 2).

[0007] In NeX, MPIs are typically arranged in a frustum shape in three-dimensional space so as to cover the distance from the camera that captured the scene to objects far away. NeX then performs radiance field learning by optimizing the pixel values ​​of the MPIs so that the radiance values ​​obtained by volume rendering the MPIs approach the radiance values ​​of the captured GT image.

[0008] This method has the advantages that, while the viewing direction is limited to the front direction of the viewing frustum that constitutes the MPI, real-time rendering is possible using general alpha compositing, the high-frequency components of the texture can be reconstructed by using a high-resolution MPI, and learning time can be shortened by correcting the values ​​of each pixel of the MPI.

[0009] However, while using NeX achieved a significant improvement of about 3,000 times in rendering speed compared to NeRF, it was unable to fully improve on issues such as the tendency for high-frequency components in textures to be lost and the need for long learning times.

[0010] In response to this, a technology has been proposed that aims to solve the above-mentioned lingering problems based on residual information between the reconstruction result using the learning result of the radiance field and the real viewpoint image (see, for example, Non-Patent Document 3).

[0011] This technology calculates the difference between a rendered image created using the learning results and a captured GT image as a residual.The residual information from multiple viewpoints is then learned using another neural network, and a light field composed only of the residual information is learned.

[0012] With this technology, during actual viewpoint reconstruction, reconstruction using the learning results of the initial radiance field is performed in addition to reconstruction using the learning results of the residual information, and by adding both, high-quality free-viewpoint reconstruction is achieved. In this case, by completing the learning of the initial radiance field early and adopting a highly efficient learning method for learning the residual information, it is possible to reduce the total learning time. Furthermore, a general-purpose method that is independent of the learning method for the initial radiance field can be used for generating and learning the residual information.

[0013] Mildenhall, B. "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis". European conference on computer vision. 2020.Wizadwongsa, Suttisak, et al. "Nex: Real-time view synthesis with neural basis expansion". Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021.Xuejian Rong, et al. "Boosting View Synthesis with Residual Transfer”.

[0014] However, the technology disclosed in Non-Patent Document 3 has two major problems. One is an increase in rendering time. That is, during rendering, the rendering process for reconstructing residual information increases compared to the rendering process using the learned radiance field, which causes a loss of real-time performance. The other is ensuring image quality performance. In general deep learning, high-frequency components of textures tend to be lost in the initial learning of the radiance field. For this reason, it is difficult to ensure image quality performance using deep learning alone.

[0015] Therefore, the present disclosure proposes an information processing device, an information processing method, and a computer-readable non-transitory storage medium that can shorten the processing time and ensure image quality performance in learning-based NVS.

[0016] In order to solve the above problem, an information processing device according to one embodiment of the present disclosure includes a learning unit and a correction unit. The learning unit learns a radiance field based on a multi-viewpoint image including a plurality of base-viewpoint images. The correction unit corrects the learned radiance field based on a base viewpoint that is a spatial correction reference, the base viewpoint image as a GT (Ground Truth) image corresponding to the base viewpoint, and a discrete three-dimensional sampling reference.

[0017] 1 is a diagram illustrating an overview of an information processing method according to an embodiment of the present disclosure. FIG. 2 is a flowchart illustrating a processing procedure of a technique disclosed in Non-Patent Document 2. FIG. 3 is a supplementary explanatory diagram (part 1) of a technique disclosed in Non-Patent Document 2. FIG. 4 is a flowchart illustrating a processing procedure of a technique disclosed in Non-Patent Document 3. FIG. 5 is a block diagram illustrating a basic configuration example of an information processing apparatus according to a first embodiment. FIG. 6 is an explanatory diagram (part 1) of information processing according to the first embodiment. FIG. 7 is an explanatory diagram (part 2) of information processing according to the first embodiment. FIG. 8 is a flowchart illustrating a processing procedure executed by an information processing apparatus according to the first embodiment. FIG. 9 is a schematic diagram of sampling points in volume rendering using MPI. FIG. 10 is a diagram illustrating a processing result by information processing according to the first embodiment. FIG. 11 is a block diagram illustrating a configuration example of an information processing apparatus according to the first embodiment. FIG. 12 is an explanatory diagram (part 1) of information processing according to a second embodiment. FIG. 13 is an explanatory diagram (part 3) of information processing according to the second embodiment. FIG. 14 is a block diagram illustrating a configuration example of an information processing apparatus according to the second embodiment. FIG. 15 is a block diagram illustrating a configuration example of a correction control value calculation unit. FIG. 16 is a flowchart illustrating a processing procedure executed by an information processing apparatus according to the second embodiment. FIG. 10 is a flowchart (part 2) showing a processing procedure executed by an information processing device according to a second embodiment. FIG. 11 is a diagram showing an example of calculation of a correction control value based on edge information. FIG. 12 is a diagram showing an example of a depth image, a detected edge, and a correction control value based on actual sample data. FIG. 13 is a diagram showing a processing result by information processing according to the second embodiment. FIG. 14 is a block diagram showing an example of the configuration of an information processing device according to a third embodiment. FIG. 15 is a flowchart showing a processing procedure executed by an information processing device according to the third embodiment. FIG. 16 is a hardware configuration diagram showing an example of a computer that realizes the functions of an information processing device.

[0018] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0019] In the following, an information processing device 100 according to a first embodiment, an information processing device 100A according to a second embodiment, and an information processing device 100B according to a third embodiment will be taken as examples of information processing devices according to an embodiment of the present disclosure (hereinafter referred to as "the present embodiment" as appropriate). An information processing method according to an embodiment of the present disclosure is an information processing method executed by these information processing devices 100, 100A, and 100B.

[0020] The present disclosure will be described in the following order: 1. Overview 1-1. Overview of information processing method according to embodiments of the present disclosure 2. First embodiment 2-1. Basic configuration example of information processing device 2-2. Description of information processing according to first embodiment 3. Second embodiment 3-1. Description of information processing according to second embodiment 4. Third embodiment 4-1. Description of information processing according to third embodiment 5. Modified example 6. Hardware configuration 7. Conclusion

[0021] <<1. Overview>> <1-1. Overview of Information Processing Method According to Embodiment of the Present Disclosure> First, an overview of the information processing method according to the present embodiment will be described using Fig. 1. Fig. 1 is a diagram illustrating an overview of the information processing method according to the embodiment of the present disclosure. Note that Fig. 1 includes an information processing method according to the prior art, and differences from the prior art correspond to steps S1, S2, and S3 in the figure.

[0022] The process will be described from a general perspective. As shown in FIG. 1, the information processing method according to this embodiment is a technology for generating a new viewpoint image (hereinafter referred to as an "NVS image") viewed from an arbitrary viewpoint in a three-dimensional space in which a subject object exists, by deep learning using a multi-view image and its camera parameters. Here, the multi-view image includes a group of GT images captured from various angles by a camera 50, and represents a plurality of input images each corresponding to a different viewpoint. In the present disclosure, the input image as the multi-view image is not limited to raw data of the captured image, but may also include images obtained by processing the raw data.

[0023] As shown in Fig. 1, in the information processing method according to this embodiment, radiance field learning is performed by deep learning using multi-viewpoint images as a dataset, and the learning results are obtained. The learning results in Fig. 1 include a radiance field (trained model) that has been trained so that, when a viewing direction from an arbitrary viewing position is input, an NVS image seen from this viewing direction can be estimated, or an MPI obtained from the trained radiance field. In the present disclosure, the trained radiance field and MPI may be simply referred to as the "learning result."

[0024] Next, in the information processing method according to this embodiment, when an arbitrary viewing direction is input, the learned radiance field is converted into a two-dimensional image using a rendering method called volume rendering, and an NVS image corresponding to the viewing direction is generated. The arrows in the NVS image shown in Figure 1 schematically represent how the position and orientation of an object (here, a cat) appear to change depending on the viewing position and viewing direction. The generated NVS image is output to a display device such as a head-mounted display or a stereoscopic display.

[0025] The general flow up to this point is also the same in the conventional technology. Prior to a more specific description of the information processing method according to this embodiment, the information processing method according to the conventional technology will be described with reference to FIGS. 2 to 5.

[0026] 2 is a flowchart showing the processing procedure of the technique disclosed in Non-Patent Document 2. FIG. 3 is a supplementary explanatory diagram (part 1) of the technique disclosed in Non-Patent Document 2. FIG. 4 is a supplementary explanatory diagram (part 2) of the technique disclosed in Non-Patent Document 2. FIG. 5 is a flowchart showing the processing procedure of the technique disclosed in Non-Patent Document 3.

[0027] In the technology disclosed in Non-Patent Document 2, i.e., "NeX," as shown in FIG. 2, the inside / outside (internal / external) camera parameters of each camera 50 are estimated using captured multi-viewpoint images and a technology such as SfM (Structure from Motion) (step S101), and the learning process of the radiance field is started using these as input.

[0028] In the learning process, a ray (one pixel in the multi-viewpoint image) is randomly selected from the input multi-viewpoint image (step S102), and the brightness value corresponding to the ray is calculated by volume rendering using the latest MPI value (step S103).

[0029] Then, based on the error between the brightness value estimated by volume rendering and the brightness value of the input image, the MPI is updated by the backpropagation algorithm (step S104).

[0030] The above MPI update process is performed a sufficient number of times, and based on the number of times, it is determined whether the learning has converged (step S105). If the learning has not converged (step S105, No), the learning process is repeated from step S102. On the other hand, if the learning has converged (step S105, Yes), the learning process is terminated and the MPI is saved (step S106).

[0031] Note that FIG. 3 is an image diagram of an MPI. As shown in FIG. 3, an MPI typically has a structure consisting of a group of layers in which semi-transparent RGBA images are stacked in a frustum shape in three-dimensional space so as to cover the entire scene, from the object closest to the camera 50 that captured the scene to a sufficiently distant object. A new viewpoint image is generated in response to a shift in viewpoint by overlaying the RGBA images that make up the MPI for each viewpoint in the viewing direction. Note that the first viewpoint in FIG. 3 is, for example, the shooting position of the camera 50. The second viewpoint is, for example, a new arbitrary viewing position that is not the shooting position of the camera 50.

[0032] The radiance field in NeX is trained so that the RGB values ​​obtained by volume rendering these MPIs are optimized to approximate the RGB values ​​captured by the camera 50.

[0033] As shown in Figure 4, the rendering process of NeX traces a ray from the viewing direction, and multiplies the transmittance α at the point where the ray intersects with the MPI by the color of that point and adds the result. ０ is color information that does not change depending on the angle at which the image is viewed, i.e., is color information that is independent of the viewing direction. １ ~k Nis color information that changes depending on the viewing angle, i.e., depends on the viewing direction.

[0034] As a result of the learning process shown in FIG. 2, any one point (here, the point in the upper right corner) of a certain layer in the MPI is calculated by the above k ０ and this k ０ k when viewed from various angles １ , ..., k Ｎ The color information is composed of:

[0035] Then, the basis function H learned from the neural network １ , ..., H Ｎ depends on the viewing angle. １ , ..., k Ｎ The weights are changed and linearly combined. This determines the RGB values ​​at the aforementioned point when viewed from a certain angle. This type of modeling makes it possible to reproduce the phenomenon where the same location on an object looks different depending on the angle.

[0036] Next, a description will be given of the processing procedure of the technique disclosed in Non-Patent Document 3, which improves on the example in Fig. 2. In the technique disclosed in Non-Patent Document 3, as shown in Fig. 5, after multi-viewpoint images and interior / exterior camera parameters, etc. are acquired (step S201), these are used as input to execute the learning process of the radiance field shown in Fig. 2 (step S202).

[0037] Then, based on the learning results, real viewpoint images (GT images) corresponding to each camera 50 of the multi-viewpoint image that was the input image are rendered (step S203). Note that in the NVS image that is the rendering result here, deviations from the input image occur depending on the estimation accuracy based on the learning results.

[0038] Therefore, in the technology disclosed in Non-Patent Document 3, a difference image between the input GT image and NVS image is calculated as residual information (step S204), and this residual information is used as input to learn the residual ray field (step S205). The learning results are saved separately from the learning results of the radiance field. Note that the difference between the radiance field and the ray field here is whether or not shape information (occupancy) is included in the output information.

[0039] The input information is common to both the radiance field and the light field, and consists of coordinate information (x, y, z) of a point in space to be estimated and three-dimensional viewing direction information (Θ, φ) from which that point is viewed.

[0040] In contrast, the output information differs between the radiance field and the light field. The output of the radiance field is color information in the viewing direction and density information (Occupancy: σ) of the point. The density information σ represents the likelihood indicating the volume density of the predicted point, or the probability of the point's existence. The relationship between the above-mentioned "shape information" or density information σ and the transmittance α shown in Figure 4 is expressed as "σ = 1 - α." In other words, the density information σ corresponds to the inverted value of the transmittance α (i.e., σ is the 1's complement of α), with "σ = 0" representing a completely transparent object and "σ = 1" representing an opaque object.

[0041] In contrast, the light field outputs only the luminance information of each viewpoint, which does not have shape information. Therefore, when rendering using the learning results from the processing procedure in Figure 5, rendering is performed using the learned radiance field as well as the residual light field information, and the final NVS image is calculated by adding the two together.

[0042] Therefore, when the learning results in Figure 5 are used compared to when the learning results in Figure 2 are used, the amount of processing required to estimate the residual ray field during rendering increases. Furthermore, with this method, the accuracy of the NVS image depends on both the radiance field and the residual ray field. Therefore, a trade-off between image quality and real-time performance arises, depending on the cost of estimating the residual ray field.

[0043] Returning to the explanation of Fig. 1, in the information processing method according to this embodiment, after learning the radiance field as shown in Fig. 1, the learning result is directly corrected based on the residual as a pre-processing for rendering (step S1). Furthermore, the region to be corrected in step S1 is limited (step S2). Furthermore, the frequency band to be used for learning is optimized (step S3).

[0044] Note that the learning results of the radiance field learning before correction, which are the subject of correction in this pre-processing, will be hereinafter referred to or illustrated as "intermediate data" as appropriate. Furthermore, the "direct correction" in step S1 refers to directly updating this intermediate data.

[0045] There are two types of learning results: an implicit representation represented by the coefficients of the neural network itself (i.e., a continuous generalized function expressed by MLP), and an explicit representation based on a discrete three-dimensional sampling standard such as MPI. Traditionally, implicit representations have been the mainstream for learning results in NeRF-based technologies, even among NVSs. In contrast, the above-mentioned NeX and other technologies perform pre-calculation by converting the learning results to a format such as MPI after learning using MLP, thereby reducing the computational load during rendering and achieving real-time rendering. On the other hand, this embodiment is characterized by optimizing the learning results obtained once in a complementary manner by directly updating intermediate data in an explicit representation such as MPI in pre-processing without using MLP.

[0046] Below, a more detailed description will be given of configuration examples of information processing devices 100, 100A, and 100B to which the information processing method according to this embodiment is applied. Note that step S1 above will be described as a first embodiment, step S2 above will be described as a second embodiment, and step S3 above will be described as a third embodiment.

[0047] <<2. First embodiment>> <2-1. Basic configuration example of information processing device> Fig. 6 is a block diagram showing a basic configuration example of an information processing device 100 according to the first embodiment. Note that in Fig. 6 and Figs. 12, 16, 17, and 23 shown later, only the components necessary for explaining this embodiment are shown in functional blocks, and descriptions of general components are omitted.

[0048] In other words, the components shown in Figures 6, 12, 16, 17, and 23 are conceptual functional components and do not necessarily have to be physically configured as shown. For example, the specific form of distribution and integration of each block is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0049] In addition, in the description using FIGS. 6, 12, 16, 17, and 23, the description of components that have already been described may be simplified or omitted.

[0050] In addition, in FIG. 6, the information processing device 100 is taken as an example, but the basic configuration example is the same for information processing devices 100A and 100B described later.

[0051] As shown in FIG. 6, the information processing device 100 includes an input unit 101 , an output unit 102 , a communication unit 103 , a storage unit 104 , and a control unit 105 .

[0052] The input unit 101 is a component that inputs data to the information processing device 100. The input unit 101 is realized by operation components such as a keyboard and a mouse, and accepts data input by a user.

[0053] The input unit 101 is realized as an input interface from various devices, for example, and accepts input of multi-viewpoint images captured by an RGB camera such as a smartphone or a single-lens camera.

[0054] The input unit 101 may be realized by various sensors such as a depth sensor, a stereo camera, or a LiDAR (Light Detection and Ranging) sensor, and may receive input of a depth image or the like.

[0055] The output unit 102 is a component that outputs data from the information processing device 100. The output unit 102 is realized by, for example, a head-mounted display or a stereoscopic display, and displays various information related to the learning results, such as NVS images rendered using the learning results.

[0056] The communication unit 103 is realized by, for example, a network adapter, and communicates with the external device 200. The external device 200 is a device, such as a game engine, that performs physical calculations based on data transmitted from the information processing device 100. The communication unit 103 transmits, for example, various pieces of information related to the learning results described above to the external device 200.

[0057] The storage unit 104 is realized by, for example, a semiconductor memory element such as a random access memory (RAM), a read only memory (ROM), or a flash memory, or a storage device such as a hard disk or an optical disk.

[0058] The control unit 105 is a so-called controller, and is realized by, for example, a GPU (Graphics Processing Unit), a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc., executing a program according to this embodiment (not shown) stored in the storage unit 104 using RAM as a work area. The control unit 105 can also be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0059] The control unit 105 has various processing units shown in FIGS. 12, 16, 17, and 23, and realizes or executes the information processing functions and actions described below while appropriately using the storage unit 104.

[0060] <2-2. Description of Information Processing According to First Embodiment> Next, information processing according to the first embodiment will be described. The information processing according to the first embodiment is a process of "directly correcting the learning result based on the residual" described as step S1 in FIG. 1. FIG. 7 is an explanatory diagram (part 1) of information processing according to the first embodiment. Also, FIG. 8 is an explanatory diagram (part 2) of information processing according to the first embodiment.

[0061] As shown in Figure 7, when a GT image is rendered using the learning results of the radiance field according to the above-mentioned conventional technology, the generated NVS image may have extremely blurred image quality in low-contrast areas, such as the wood grain of a desk, compared to the GT image. The right diagram in Figure 7 shows the difference between the GT image and the NVS image.

[0062] In order to improve this, and in addition to shortening the processing time, the information processing according to the first embodiment directly corrects the learning result of the radiance field, i.e., the above-mentioned MPI. Specifically, as shown in Fig. 8, in the information processing method according to the first embodiment, after the conventional radiance field learning shown in Fig. 2 etc. and before rendering, the pre-processing according to this embodiment is performed on the intermediate data.

[0063] In this pre-processing, after learning the radiance field, a base viewpoint, which is a spatial correction reference, is determined / selected, and the residual between a base viewpoint image corresponding to the base viewpoint and an NVS image corresponding to this base viewpoint reconstructed using the learning results is calculated. While the residual is expressed as a difference using a subtractor in FIG. 8 , the residual may also be a ratio. In this case, the image used for correction may be the original image used in learning, or a processed image of the original image. The processed image may include, for example, an image whose resolution has been adjusted according to the viewing distance. This can suppress the effect of changes in resolution according to the viewing distance during correction, thereby improving the quality of the corrected NVS image.

[0064] 8 shows an example in which one GT image is selected from multiple input images for each correction, but this embodiment is not limited to processing in which one input image is used as the GT image. For example, if the desired processing time can be reduced, an edited image of an original image having the same viewpoint may be additionally selected as the GT image. By using a combination of the original image and the edited image, for example, it is possible to adjust the taste of the NVS image obtained from the learning results after correction.

[0065] The base viewpoint may be arbitrarily specified by the user or may be automatically determined by the information processing device 100. When automatically determined, the information processing device 100 may determine the base viewpoint based on, for example, the front direction of the subject. In addition, the base viewpoint may be determined based on, for example, the camera position directly facing the learned MPI, the center or center of gravity of the distribution of the aforementioned σ, or, if the display device is a stereoscopic display, the normal direction of the display surface. Furthermore, the apex of a quadrangular pyramid formed by extending the side edges of the learned MPI may be set as the camera position.

[0066] In this pre-processing, the calculated residual is multiplied as a correction value to correct the MPI so that the low-quality MPI resulting from the learning becomes high-quality. At this time, in this pre-processing, this correction value is used to correct the color information in the learning result. Note that although FIG. 8 shows the correction to the target value by a multiplier, correction by addition, subtraction, or the like may also be used.

[0067] 9 is a flowchart showing the processing procedure executed by the information processing device 100 according to the first embodiment. As shown in FIG. 9, the control unit 105 of the information processing device 100 first acquires multi-viewpoint images and interior / exterior camera parameters, etc. (step S301), and then executes the radiance field learning process shown in FIG. 2 and other figures using these as input (step S302).

[0068] Then, the control unit 105 determines a base viewpoint from among the real viewpoints (the viewpoints at which the GT image was actually taken), performs volume rendering of this base viewpoint image using the learning results (step S303), and generates an NVS image.

[0069] The volume rendering in step S303 is performed using, for example, the following equations (a), (b), and (c): ｎ is the brightness information of the nth point, and ρ ｎ indicates the contribution rate of the nth point. In addition, in formula (b), σ indicates the aforementioned existence probability "Occupancy," and α indicates the aforementioned transmittance. Formula (b) is derived from the aforementioned relationship between σ and α, "σ = 1 - α."

[0070]

[0071]

[0072]

[0073] 10 is a schematic diagram of sampling points in volume rendering using MPI. Here, the subscript i corresponds to the layer ID when counting the MPI layers from the front. Note that MPI is an example of a sampling standard in this disclosure, and has a position and orientation independent of the position and orientation of the input image.

[0074] Returning to the explanation of Fig. 9, the control unit 105 then compares the GT image and the NVS image, which are the targets of reconstruction, and calculates a correction value (step S304). The formulas for calculating the correction value for the n-th point of volume rendering are shown in the following formulas (d) and (e). Note that I in formula (e) ＧＴ is the brightness value of the GT image, and I ＮＶＳ indicates the predicted value using the learning results.

[0075]

[0076]

[0077] Finally, the control unit 105 corrects the MPI using this correction value, specifically, corrects the color information of the learned MPI (step S305). The correction formula used at this time is shown in formula (f). Note that g in formula (f) represents the correction adjustment gain. In the first embodiment, no adjustment is made using the correction adjustment gain, so it is sufficient to set it to a fixed value of 1.0.

[0078]

[0079] Fig. 11 is a diagram showing the processing results of the information processing according to the first embodiment. As shown in Fig. 11, the information processing shown in Fig. 10 makes it possible to obtain a rendering result with higher resolution (see "Correction On" in the figure) compared to the conventional technology (see "Correction Off" in the figure).

[0080] It should be noted that, as color information to be corrected in the information processing according to the first embodiment, it is desirable to correct a term of the learned color information of the radiance field that is not dependent on the viewing position / viewing direction at the time of rendering, that is, a term that is independent of changes in the viewing position / viewing direction of the new viewpoint image. ０ For example, in MPI in NeX, this is color information called "mpi_c," and in expressions using spherical harmonic functions, this corresponds to an offset term that is independent of the viewing position / viewing direction. This makes it possible to expect a robust correction effect even when the viewing position moves, making it possible to create high-quality new viewpoint images.

[0081] A block diagram is shown in Fig. 12. Fig. 12 is a block diagram showing an example of the configuration of the information processing device 100 according to the first embodiment. As shown in Fig. 12, the information processing device 100 (its control unit 105) has a radiance field learning unit 105b, a rendering unit 105d, a correction value calculation unit 105e, and a radiance field correction unit 105g.

[0082] The radiance field learning unit 105b executes the learning process corresponding to the above-mentioned step S302 based on the multi-viewpoint images and the camera parameters. The rendering unit 105d executes the rendering process corresponding to the above-mentioned step S303 based on the learning results.

[0083] The correction value calculation unit 105e performs the correction value calculation process corresponding to the above-mentioned step S304 based on the estimated real viewpoint image (i.e., the NVS image) and the base viewpoint image. The radiance field correction unit 105g performs the radiance field correction process corresponding to the above-mentioned step S305 based on the learning result and the correction value data, and outputs the corrected / improved learning result.

[0084] <<3. Second Embodiment>> <3-1. Description of Information Processing According to Second Embodiment> Next, information processing according to the second embodiment will be described. The information processing according to the second embodiment is the processing of "limiting the correction target area" described as step S2 in FIG. 1. FIG. 13 is an explanatory diagram (part 1) of information processing according to the second embodiment. Also, FIG. 14 is an explanatory diagram (part 2) of information processing according to the second embodiment. Also, FIG. 15 is an explanatory diagram (part 3) of information processing according to the second embodiment.

[0085] The information processing method according to the first embodiment described above performs correction on the entire learned radiance field, which may result in a similar correction being applied to occlusion regions that are not actually visible from the base viewpoint, resulting in overcorrection.

[0086] Figure 13 shows an example of the positional relationship between the base viewpoint camera, MPI, and subject. In the example of Figure 13, only the round object in the foreground is visible from the base viewpoint camera, and the square object in the background is not visible. Therefore, correcting the square object in the background using information from the GT image from the base viewpoint camera results in overcorrection.

[0087] Overcorrection is not very noticeable in NVS images from viewpoints near the base viewpoint, but this overcorrection becomes very noticeable from viewpoints that are far away from the base viewpoint.

[0088] Figure 14 shows a GT image from the base viewpoint, and Figure 15 shows an example of overcorrection of the learning results in MPIs arranged in a frustum shape from the base viewpoint. For convenience, Figure 14 is shown as a schematic line drawing, but it is assumed to be a GT image captured of a scene with a bronze statue in the center against a building in the background. Furthermore, the MPI layer shown in Figure 15 corresponds to a layer located quite close to the building in the depth direction from the base viewpoint.

[0089] 15 in comparison with Fig. 14, it can be seen that the outline of the bronze statue in the foreground appears on the building in the background as a false outline due to overcorrection, as shown in part M1 in the figure. Therefore, in the information processing according to the second embodiment, processing is added to limit the area to be corrected so as to suppress overcorrection.

[0090] Fig. 16 is a block diagram showing an example of the configuration of an information processing device 100A according to the second embodiment. Fig. 17 is a block diagram showing an example of the configuration of a correction control value calculation unit 105f. Note that Fig. 16 corresponds to Fig. 12 used in the description of the first embodiment, and therefore differences from the first embodiment will be mainly described here.

[0091] 16 , the information processing apparatus 100A (controller 105) according to the second embodiment differs from the first embodiment in that it further includes a correction control value calculator 105f (which is an example of a “calculator”). The correction control value calculator 105f calculates a correction control value based on the σ of the MPI, and executes calculation processing of the correction control value to be output to the radiance field corrector 105g.

[0092] 17 , the correction control value calculation unit 105f includes an edge detection unit 105fa, a visibility determination unit 105fb, and a multiplier 105fc. The edge detection unit 105fa performs edge detection processing to detect object boundaries based on σ. The visibility determination unit 105fb performs visibility determination processing to calculate a control value based on the visibility of an object from a base viewpoint based on σ. The multiplier 105fc multiplies the results of the edge detection processing and the visibility determination processing to obtain a final control value and output it.

[0093] A more detailed description will be given below. Fig. 18 is a flowchart (part 1) showing the processing procedure executed by the information processing device 100A according to the second embodiment. Fig. 19 is a flowchart (part 2) showing the processing procedure executed by the information processing device 100A according to the second embodiment.

[0094] The process up to the calculation of the correction value in step S403 in FIG. 18 is the same as that in FIG. 9 already shown, and therefore the description of steps S401 to S403 will be omitted here.

[0095] Following the calculation of the correction value in step S403, the correction control value calculation unit 105f executes calculation of a correction control value (step S404). Note that, hereinafter, the correction control value may be abbreviated as "control value."

[0096] In the process of calculating the correction control value, as shown in FIG. 19, the visibility determining unit 105fb calculates the control value based on the visibility of the sampling points (step S501).

[0097] The visibility determination unit 105fb first calculates the contribution rate ρ of each sampling point using the σ information of the MPI by the above formula (c). Here, if ρ is close to 0 and sufficiently small, it means that the object is transparent or is occluded by a foreground object and cannot be seen from the base viewpoint.

[0098] The visibility determination unit 105fb calculates a control value related to visibility using the following formula (g): m ｎ indicates a control value for determining visibility, and th indicates a threshold for determining whether visibility is acceptable or not. th may be set to a sufficiently small value as appropriate. The visibility determining unit 105fb may determine a correction control value such that correction by the correction value is weakened when visibility is low, and ρ ｎ It is sufficient to determine it in a way that monotonically increases with respect to .

[0099]

[0100] Next, the edge detection unit 105fa calculates a control value based on the depth information. For example, in "NeRF," σ tends to be widely distributed near the boundary between foreground and background objects in order to represent occlusion that occurs with changes in viewpoint, and tends to be widely distributed across the foreground and background.

[0101] For this reason, if correction is performed for the entire area along the light ray, the correction of the foreground will propagate to the background, which tends to cause overcorrection. This overcorrection also tends to be noticeable at viewpoints other than the base viewpoint camera, which is an obstacle to expanding the viewpoint freedom during reconstruction.

[0102] Therefore, it is effective to apply a process to weaken the correction near the edge based on the edge information of the object boundary. The edge detection unit 105fa first calculates a depth image d based on the following formula (h) (step S502). ｎ indicates the depth of the nth sampling point.

[0103]

[0104] Next, the edge detection unit 105fa calculates edges in the depth image d (step S503). As a method for calculating edges, a general method such as a Sobel filter may be used.

[0105] The edge detector 105fa then calculates a control value based on the edge information of the calculated edge (step S504). Fig. 20 is a diagram showing an example of calculation of a correction control value based on edge information.

[0106] As shown in Fig. 20, the edge detection unit 105fa calculates the correction control value so that, for example, the correction control value decreases as the edge strength increases. Note that Fig. 20 is merely an example, and it is sufficient if the control value monotonically decreases toward 0 as the edge strength increases.

[0107] 21 is a diagram showing an example of a depth image d, detected edges, and correction control values ​​based on actual sample data. The darker the portion in the correction control value of FIG. 21, the weaker the correction of the learning result.

[0108] Returning to the description of Fig. 19, the multiplier 105fc integrates the control values, for example by multiplying the control value output by the visibility determination unit 105fb and the control value output by the edge detection unit 105fa (step S505), to obtain a final corrected control value.

[0109] Although edge information of the depth image d is used here to determine the object boundary, it is also effective to use, as an alternative, a variance image of σ in the depth direction of sampling points on the ray for volume rendering where the contribution rate ρ is greater than 0. This is because this variance image also indicates that σ is widely distributed.

[0110] Returning to the description of Fig. 18, following the calculation process of the correction control value in step S404, the radiance field correction unit 105g corrects the MPI using the correction value from the correction value calculation unit 105e and the control value from the correction control value calculation unit 105f (step S405).

[0111] At this time, the radiance field correction unit 105g corrects the color information of the learned radiance field based on, for example, the following equation (i).

[0112]

[0113] An example of a radiance field corrected by the above-described information processing according to the second embodiment is shown in Fig. 22. Fig. 22 is a diagram showing the processing results of the information processing according to the second embodiment.

[0114] The left diagram in Fig. 22 is the same as the overcorrection data already shown in the right diagram in Fig. 15. As shown in Fig. 22, when correction is performed with a limited correction area by the information processing according to the second embodiment (when correction control is performed), it can be seen that overcorrection is suppressed compared to when correction control is not performed (see part M2 in the diagram).

[0115] <<4. Third embodiment>> <4-1. Description of information processing according to the third embodiment> Next, information processing according to the third embodiment will be described. The information processing according to the third embodiment is processing for "optimizing frequency bands to be used for learning," which is described as step S3 in FIG. 1 .

[0116] It is generally known that in NeRF-based learning, learning of color information begins with low-frequency components, and as learning continues for a long time, learning progresses to high-frequency components. It is also known that the higher the frequency, the longer the time required for convergence increases nonlinearly. Therefore, by terminating radiance field learning at an early stage and combining it with the information processing method according to this embodiment, the learning time can be shortened.

[0117] It is also known that the learning time generally increases in proportion to the resolution of the input image. Therefore, the learning time can be further reduced by concentrating the radiance field learning on the low-frequency components and supplementing the high-frequency components with the information processing method according to this embodiment.

[0118] Fig. 23 is a block diagram showing an example of the configuration of an information processing device 100B according to the third embodiment. Fig. 24 is a flowchart showing a processing procedure executed by the information processing device 100B according to the third embodiment. Note that Fig. 23 corresponds to Fig. 16 used in the description of the second embodiment, and therefore differences from the second embodiment will be mainly described here.

[0119] 23 , the information processing device 100B (controller 105) according to the third embodiment differs from the second embodiment in that it further includes a reduction unit 105 a and an enlargement unit 105 c. In the present disclosure, the reduction unit 105 a and the enlargement unit 105 c that perform resolution conversion may be referred to as a resolution conversion unit.

[0120] The reduction unit 105a reduces the multi-viewpoint images by converting their resolution and calculates camera parameters according to the reduction ratio before the radiance field learning process. The enlargement unit 105c enlarges each radiance field by converting its resolution to the resolution of the original image before reduction after the radiance field learning process.

[0121] 24, the control unit 105 of the information processing device 100B first acquires a multi-view image and internal / external camera parameters (step S601), and then the reduction unit 105a reduces the input image and calculates camera parameters according to the reduction rate (step S602).

[0122] Next, the radiance field learning unit 105b executes the learning process of the radiance field shown in FIG. 2 and the like, using the reduced image of the low-frequency component and the camera parameters corresponding thereto as input (step S603).

[0123] The enlargement unit 105c then enlarges each MPI to the resolution of the original image before reduction (step S604). Specifically, the enlargement unit 105c enlarges each layer information of the MPI using conventional 2D image enlargement processing to restore high-frequency components. Thereafter, the enlarged layer information is used to perform the same correction processing as in the second embodiment (steps S605 to S608).

[0124] The key point here is that a GT image is used to calculate the correction value. Generally, in processes that reduce and then enlarge an image, artifacts due to aliasing occur because high-frequency components lost in the reduction process are forcibly restored in the enlargement process.

[0125] In contrast, in the third embodiment, by using a high-resolution original image as a GT image, in addition to correcting degradation due to learning, aliasing due to enlargement can also be directly corrected based on the residual error according to this embodiment.

[0126] Here, the correction value and correction control value are calculated after expanding the MPI obtained from the learned radiance field, but the order of expansion and calculation may be reversed, taking into account the trade-off between computational effort and performance.

[0127] <<5. Modifications>> Although the embodiments of the present disclosure have been described, the information processing method according to the present embodiments can also be modified in other ways.

[0128] For example, although the above description has been given using radiance field learning using MPI, other scene representations that represent discrete, three-dimensional sampling standards other than MPI may also be used. Examples of other representation methods that can be used include voxel grids, tensor decomposition, and hash tables. Similar effects can be achieved with these representation methods by applying corrections to the corresponding color information, particularly information that is independent of the viewing direction.

[0129] In addition, although the present embodiment has been described using an example in which only one viewpoint is used as the base viewpoint, correction may be performed using information from multiple viewpoints. In this case, MPIs corrected for multiple real viewpoints may be prepared, and the MPIs may be switched or interpolated between MPIs depending on the viewing direction input during rendering. Alternatively, correction may be performed using separate GT images for each sampling point in space, taking visibility into consideration.

[0130] Furthermore, when volume rendering a base viewpoint image using the learning results, rendering may be performed for multiple different viewing directions, and the average value or the like of the residuals with each generated NVS image may be used to calculate the correction value.

[0131] The GT image may be the image itself captured by the camera 50, or may be an image created in advance by 2D processing. For example, the GT image may be created by processing the image to emphasize the texture of hair and smooth the skin, and then used to find the residual.

[0132] Furthermore, the signals to be corrected can be limited. For example, all RGB channels can be processed as color information, or only luminance information such as Y or L can be processed after conversion to a color space such as YUV or Lab. Generally, the human eye has high sensitivity to the spatial resolution of luminance information and low sensitivity to the spatial resolution of color information. This characteristic can be utilized to reduce the amount of processing.

[0133] Furthermore, among the processes described in the above-described embodiments of the present disclosure, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0134] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0135] For example, the correction value calculation unit 105e, the correction control value calculation unit 105f, the radiance field correction unit 105g, etc. may be integrated into one processing unit.

[0136] The above-described embodiments of the present disclosure can be combined as appropriate within the scope of the present disclosure without causing any inconsistency in the processing content. The order of the steps shown in the sequence diagrams or flowcharts of the present embodiments can be changed as appropriate.

[0137] <<6. Hardware Configuration>> Furthermore, the information processing devices 100, 100A, and 100B according to the above-described embodiments of the present disclosure are realized by, for example, a computer 1000 configured as shown in FIG. 25 . FIG. 25 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing devices 100, 100A, and 100B. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a secondary storage device 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0138] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the secondary storage device 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the secondary storage device 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0139] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0140] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records the program according to this embodiment, which is an example of the program data 1450.

[0141] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550. For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0142] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0143] For example, when the computer 1000 functions as the information processing devices 100, 100A, and 100B, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to realize the functions of the control unit 105. The secondary storage device 1400 stores the program according to this embodiment, various models used in the information processing according to this embodiment, and various data as appropriate. The CPU 1100 reads and executes program data 1450 from the secondary storage device 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0144] <<7. Conclusion>> As described above, according to one embodiment of the present disclosure, the information processing devices 100, 100A, and 100B include a radiance field learning unit 105b (equivalent to an example of a "learning unit") and a radiance field correction unit 105g (equivalent to an example of a "correction unit"). The radiance field learning unit 105b learns a radiance field based on a multi-viewpoint image including a plurality of base-viewpoint images. The radiance field correction unit 105g corrects the learned radiance field based on a base viewpoint that is a spatial correction reference, the base-viewpoint image as a GT image corresponding to the base viewpoint, and a discrete three-dimensional sampling reference. This makes it possible to reduce the processing time and ensure image quality performance in learning-based NVS.

[0145] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0146] Furthermore, the effects of each embodiment described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0147] Note that the present technology may also have the following configuration. (1) An information processing device including: a learning unit that learns a radiance field based on a multi-viewpoint image including a plurality of base-viewpoint images; and a correction unit that corrects the learned radiance field based on a base viewpoint that is a spatial correction standard, the base viewpoint image as a GT (Ground Truth) image corresponding to the base viewpoint, and a discrete three-dimensional sampling standard. (2) The information processing device according to (1), wherein the base viewpoint corresponds to one of the viewpoints of the multi-viewpoint image. (3) The information processing device according to (2), wherein the correction unit corrects the learned radiance field based on a correction value corresponding to a residual between the base viewpoint image and a new viewpoint image corresponding to the base viewpoint that is generated using the learned radiance field. (4) The information processing device according to (3), wherein the correction unit limits a correction target region of the learned radiance field so as to suppress overcorrection using the correction value. (5) The information processing device according to (4), further comprising: a calculation unit that calculates a correction control value that limits the correction target region based on the visibility of the object from the base viewpoint in the learned radiance field and edge information related to the edges of the object, wherein the correction unit corrects the learned radiance field using the correction control value calculated by the calculation unit and the correction value. (6) The information processing device according to (5), wherein the calculation unit calculates the correction control value so that correction by the correction value is weakened when the visibility is low. (7) The information processing device according to (5) or (6), wherein the calculation unit calculates the correction control value so that correction by the correction value is weakened as the edge strength included in the edge information increases. (8) The information processing device according to (2), wherein the sampling reference includes multiplane images (MPIs) in a view frustum shape, and the apex of a quadrangular pyramid formed by extending the sides of the MPI corresponds to the base viewpoint. (9) The information processing device according to (8), wherein the correction unit corrects color information of the learned radiance field using the MPI.(10) The information processing device according to (9), wherein the correction unit corrects the MPI term representing the color information independent of a change in position of a new viewpoint image generated using the corrected radiance field. (11) The information processing device according to any one of (1) to (10), further comprising a resolution conversion unit that performs resolution conversion by enlarging or reducing an image, wherein the resolution conversion unit reduces the multi-viewpoint image, the learning unit learns the radiance field based on the reduced multi-viewpoint image, the resolution conversion unit reduces the multi-viewpoint image, and the correction unit corrects the enlarged radiance field based on the multi-viewpoint image before the reduction. (12) The information processing device according to (11), wherein the sampling standard includes view frustum-shaped multiplane images (MPIs), and the resolution conversion unit enlarges the radiance field by enlarging the MPI. (13) An information processing method comprising: learning a radiance field based on multi-viewpoint images including a plurality of base-viewpoint images; and correcting the learned radiance field based on a base viewpoint that is a spatial correction reference, the base viewpoint image as a GT (Ground Truth) image corresponding to the base viewpoint, and a discrete three-dimensional sampling reference. (14) A computer-readable non-transitory storage medium storing a program for causing a computer to perform the steps of: learning a radiance field based on multi-viewpoint images including a plurality of base-viewpoint images; and correcting the learned radiance field based on a base viewpoint that is a spatial correction reference, the base viewpoint image as a GT (Ground Truth) image corresponding to the base viewpoint, and a discrete three-dimensional sampling reference.

[0148] 50 Camera 100, 100A, 100B Information processing device 101 Input unit 102 Output unit 103 Communication unit 104 Memory unit 105 Control unit 105a Reduction unit 105b Radiance field learning unit 105c Enlargement unit 105d Rendering unit 105e Correction value calculation unit 105f Correction control value calculation unit 105fa Edge detection unit 105fb Visibility determination unit 105fc Multiplier 105g Radiance field correction unit 200 External device

Claims

1. An information processing device comprising: a learning unit that learns a radiance field based on a multi-viewpoint image including a plurality of base viewpoint images; and a correction unit that corrects the learned radiance field based on a base viewpoint that is a spatial correction standard, the base viewpoint image as a GT (Ground Truth) image corresponding to the base viewpoint, and a discrete three-dimensional sampling standard.

2. The information processing device according to claim 1, wherein the base viewpoint corresponds to one of the viewpoints of the multi-view image.

3. The information processing device according to claim 2, wherein the correction unit corrects the learned radiance field based on a correction value corresponding to a residual between the base viewpoint image and a new viewpoint image corresponding to the base viewpoint generated using the learned radiance field.

4. The information processing device according to claim 3, wherein the correction unit limits a correction target area of ​​the learned radiance field so as to suppress overcorrection using the correction value.

5. The information processing device of claim 4, further comprising a calculation unit that calculates a correction control value that limits the correction target area based on the visibility of the object from the base viewpoint in the learned radiance field and edge information regarding the edges of the object, wherein the correction unit corrects the learned radiance field using the correction control value and the correction value calculated by the calculation unit.

6. The information processing device according to claim 5, wherein the calculation unit calculates the correction control value so that the correction by the correction value is weakened when the visibility is low.

7. The information processing device according to claim 5, wherein the calculation section calculates the correction control value such that the correction by the correction value becomes weaker as the edge strength included in the edge information increases.

8. The information processing device according to claim 2, wherein the sampling reference includes a frustum-shaped MPI (Multiplane Image), and the apex of a quadrangular pyramid formed by extending the side edges of the MPI corresponds to the base viewpoint.

9. The information processing device according to claim 8, wherein the correction unit corrects color information of the learned radiance field using the MPI.

10. The information processing device according to claim 9, wherein the correction unit corrects a term of the MPI that represents the color information independent of a change in position of a new viewpoint image generated using the corrected radiance field.

11. An information processing device as described in claim 1, further comprising a resolution conversion unit that performs resolution conversion by enlarging or reducing an image, wherein the resolution conversion unit reduces the multi-view image, the learning unit learns the radiance field based on the reduced multi-view image, the resolution conversion unit reduces the multi-view image, and the correction unit corrects the enlarged radiance field based on the multi-view image before the reduction.

12. The information processing device according to claim 11, wherein the sampling standard includes frustum-shaped multiplane images (MPIs), and the resolution conversion unit expands the radiance field by expanding the MPIs.

13. An information processing method comprising: learning a radiance field based on a multi-viewpoint image including a plurality of base viewpoint images; and correcting the learned radiance field based on a base viewpoint that is a spatial correction standard, the base viewpoint image as a GT (Ground Truth) image corresponding to the base viewpoint, and a discrete three-dimensional sampling standard.

14. A computer-readable non-transitory storage medium storing a program for causing a computer to learn a radiance field based on a multi-viewpoint image including a plurality of base viewpoint images, and correct the learned radiance field based on a base viewpoint that is a spatial correction standard, the base viewpoint image as a GT (Ground Truth) image corresponding to the base viewpoint, and a discrete three-dimensional sampling standard.

Images