Information processing device, information processing method, and computer-readable non-transitory storage medium

Abstract

This information processing device comprises a learning unit and a light beam determination unit. The learning unit executes learning processing for learning a radiation luminance field for generating a new viewpoint image corresponding to an arbitrary viewing direction on the basis of a multi-viewpoint image including a plurality of two-dimensional images. The light beam determination unit determines whether the light beam can be used in the learning processing on the basis of a positional relationship between a light beam of interest in the learning processing and a preset spatial learning range of the radiation luminance field.

Description

Information processing device, information processing method, and computer-readable non-transitory storage medium

[0001] The present disclosure relates to an information processing device, an information processing method, and a computer-readable non-transitory storage medium.

[0002] Conventionally, an image synthesis technology known as Novel View Synthesis (NVS) is known, which captures a three-dimensional space based on multi-viewpoint images from each real viewpoint, and generates a novel viewpoint image from an arbitrary new viewpoint based on the learning results of the radiance field.

[0003] In this NVS, highly accurate reconstruction of the radiance field is important for improving the sense of immersion and realism in XR (Cross Reality) and virtual production.

[0004] As a method for reconstructing the radiance field in three-dimensional space with high accuracy, a method using a neural network called NeRF (Neural Radiance Fields) has been proposed in recent years (see, for example, Non-Patent Document 1).

[0005] NeRF represents colors that depend on the density and viewing direction of a three-dimensional space as a multilayer perceptron (MLP), and trains the MLP so that the brightness values ​​obtained by volume rendering approach the brightness values ​​of the actually captured image. However, NeRF has issues such as difficulty in real-time rendering for new input viewing directions, a tendency to lose high-frequency components in texture, and the need for long learning times.

[0006] To address this issue, a method called NeX (NeXt-level view-dependent effects) has been proposed, which arranges multi-layer images called MPI (Multiplane Images) (see, for example, Non-Patent Document 2).

[0007] In NeX, MPIs are typically arranged in a frustum shape in three-dimensional space so as to cover the entire scene from the camera that captured the scene to objects at a sufficiently distant depth. Learning of the radiance field in NeX is performed by optimizing the value of each pixel in the MPI so that the luminance values ​​obtained by volume rendering the MPIs approach the luminance values ​​of the actual captured image.

[0008] This method has the advantages that, while the viewing direction is limited to the front direction of the viewing frustum that constitutes the MPI, real-time rendering is possible using general alpha compositing, the high-frequency components of the texture can be reconstructed by using a high-resolution MPI, and learning time can be shortened by directly correcting the values ​​of each pixel of the MPI.

[0009] Mildenhall, B. "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis". European conference on computer vision. 2020. Wizadwongsa, Suttisak, et al. "Nex: Real-time view synthesis with neural basis expansion". Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021.

[0010] However, when using NeX, it is necessary to specify the resolution, position, and size of the MPI to be placed in three-dimensional space before starting learning. Here, if the resolution of the MPI (i.e., the amount of data) is fixed to a constant value, there is a trade-off between the perceived resolution of the new viewpoint image and the size of the range in which the radiance field is reconstructed.

[0011] In other words, in order to improve the perceived resolution of new viewpoint images within realistic data volume constraints, it is necessary to reduce the size of the MPI and limit the range in which the radiance field is reconstructed. The problem that arises in this case is the presence of objects outside the range of the viewing frustum in which the MPI is placed.

[0012] That is, when a ray observing a point on an object that exists outside the range of this viewing frustum is included in the input image for training, in order to reproduce the luminance value observed at this point, training is performed so that the luminance value is reproduced using other sampling points on the same ray that are included within the range of the viewing frustum. This may result in an object surface being formed at a position where no actual object surface exists, which could cause the entire training of the radiance field to fail.

[0013] Therefore, the present disclosure proposes an information processing device, an information processing method, and a computer-readable non-transitory storage medium that are capable of avoiding the failure of radiance field learning.

[0014] In order to solve the above problem, an information processing device according to one embodiment of the present disclosure includes a learning unit and a ray determination unit. The learning unit executes a learning process to learn a radiance field for generating a new viewpoint image according to an arbitrary viewing direction based on a multi-viewpoint image including a plurality of two-dimensional images. The ray determination unit determines whether a ray of interest in the learning process can be used in the learning process based on a positional relationship between a ray of interest in the learning process and a predetermined spatial learning range of the radiance field.

[0015] FIG. 1 is a diagram illustrating an overview of an information processing method according to an embodiment of the present disclosure. FIG. 2 is a flowchart illustrating a processing procedure of the technology disclosed in Non-Patent Document 2. FIG. 3 is a diagram illustrating a supplementary explanatory diagram (part 1) of the technology disclosed in Non-Patent Document 2. FIG. 4 is a diagram illustrating a cause of failure in learning of a radiance field. FIG. 5 is a block diagram illustrating an example configuration of an information processing device according to an embodiment of the present disclosure. FIG. 6 is a flowchart illustrating a processing procedure executed by an information processing device according to an embodiment of the present disclosure. FIG. 7 is an explanatory diagram of a light ray determination process. FIG. 8 is a flowchart illustrating a processing procedure of a light ray determination process. FIG. 9 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of an information processing device.

[0016] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0017] In the following, an information processing device according to an embodiment of the present disclosure (hereinafter referred to as "the present embodiment") is assumed to be information processing device 100 (see FIG. 6) that learns a radiance field in a learning-based NVS based on multi-viewpoint images captured at each real viewpoint. In the following, an information processing method according to the present embodiment is assumed to be an information processing method executed by information processing device 100.

[0018] The present disclosure will be described in the following order: 1. Overview 2. Configuration example of information processing device according to embodiment of the present disclosure 2-1. Block diagram 2-2. Information processing procedure according to embodiment of the present disclosure 3. Modification 4. Hardware configuration 5. Conclusion

[0019] <<1. Overview>> First, an overview of an information processing method according to this embodiment will be described using Fig. 1. Fig. 1 is a diagram illustrating an overview of an information processing method according to an embodiment of the present disclosure. Note that Fig. 1 includes the contents of an information processing method according to conventional technology, and differences from the conventional technology correspond to steps S1 and S2 in the figure.

[0020] The following will explain the general flow of the processing. As shown in Figure 1, the information processing method according to this embodiment is a technology for generating an NVS image, which is a new viewpoint image viewed from a new arbitrary viewpoint in a three-dimensional space in which a subject object exists, by deep learning using multi-viewpoint images including a group of two-dimensional images captured from various angles by a camera 50 and the camera parameters thereof. Here, the multi-viewpoint images as input images may include not only raw data of captured images but also images obtained by processing the raw data.

[0021] As shown in Fig. 1, in the information processing method according to this embodiment, radiance field learning is performed by deep learning using multi-viewpoint images as a dataset, and the learning results are obtained. The learning results in Fig. 1 include a radiance field (trained model) that has been trained to estimate an NVS image seen from an input viewing direction from an arbitrary viewing position, or an MPI obtained from the trained radiance field. In the present disclosure, the trained radiance field and MPI may be simply referred to as the "learning results."

[0022] Next, in the information processing method according to this embodiment, when an arbitrary viewing direction is input, the learned radiance field is converted into a two-dimensional image using a rendering method called volume rendering, and an NVS image corresponding to the viewing direction is generated. The arrows in the NVS image shown in Figure 1 schematically represent how the position and orientation of an object (here, a cat) appear to change depending on the viewing position and viewing direction. The generated NVS image is output to a display device such as a head-mounted display or a stereoscopic display.

[0023] The general flow up to this point is also the same in the conventional technology. Prior to a more specific description of the information processing method according to this embodiment, the information processing method according to the conventional technology will be described with reference to FIGS. 2 to 5.

[0024] Fig. 2 is a flowchart showing the processing procedure of the technique disclosed in Non-Patent Document 2. Fig. 3 is a supplementary explanatory diagram (part 1) of the technique disclosed in Non-Patent Document 2. Fig. 4 is a supplementary explanatory diagram (part 2) of the technique disclosed in Non-Patent Document 2. Fig. 5 is an explanatory diagram of the cause of failure in learning of a radiance field.

[0025] In the technology disclosed in Non-Patent Document 2, i.e., "NeX," as shown in FIG. 2, the inside / outside (internal / external) camera parameters of each camera 50 are estimated using captured multi-viewpoint images and a technology such as SfM (Structure from Motion) (step S101), and the learning process of the radiance field is started using these as input.

[0026] In the learning process, first, a ray (pixel) is randomly selected from the input multi-view image (step S102). Then, the brightness value corresponding to the ray is calculated by volume rendering using the latest MPI value (step S103). The latest MPI includes, for example, a temporary MPI during learning that is expanded on RAM.

[0027] Then, based on the error between the brightness value estimated by volume rendering and the brightness value of the input image, the MPI is updated by the backpropagation algorithm (step S104).

[0028] The above MPI update process is performed a sufficient number of times, and based on the number of times, it is determined whether the learning has converged (step S105). If the learning has not converged (step S105, No), the learning process is repeated from step S102. On the other hand, if the learning has converged (step S105, Yes), the learning process is terminated and the MPI is saved (step S106).

[0029] One example of a method for calculating a luminance value corresponding to a ray is to store RGB values ​​and alpha values ​​in the MPI and integrate these values ​​using alpha composite (see, for example, non-patent document "Zhou, Tinghui, et al. "Stereo magnification." ACM Transactions on Graphics 37.4 (2018): 1-12."). Furthermore, viewpoint-dependent luminance values ​​may be calculated by storing, in the MPI, feature amounts according to the direction of the ray, in addition to the RGB values ​​and alpha values.

[0030] FIG. 3 is a conceptual diagram of an MPI. As shown in FIG. 3, an MPI typically has a structure consisting of a group of layers in which semi-transparent RGBA images are stacked in a frustum shape in three-dimensional space to cover the entire scene, from the object closest to the camera 50 that captured the scene to a sufficiently distant object. An NVS image corresponding to the shift in viewpoint is generated by overlaying the RGBA images that make up the MPI for each viewpoint in the viewing direction. The first viewpoint in FIG. 3 is, for example, the shooting position of the camera 50. The second viewpoint is, for example, a new arbitrary viewing position that is not the shooting position of the camera 50.

[0031] The radiance field in NeX is trained so that the RGB values ​​obtained by volume rendering these MPIs are optimized to approximate the RGB values ​​captured by the camera 50.

[0032] In the rendering process of NeX, a ray is traced from the viewing direction, and as shown in Figure 4, the transmittance α at the point where the ray intersects with the MPI is multiplied by the color of that point and added. ０ is color information that does not change depending on the angle at which the image is viewed, i.e., is color information that is independent of the viewing direction. １ ~k N is color information that changes depending on the viewing angle, i.e., depends on the viewing direction.

[0033] As a result of the learning process shown in FIG. 2, any one point (here, the point in the upper right corner) of a certain layer in the MPI is calculated by the above k ０ and this k ０ k when viewed from various angles １ , ..., k Ｎ The color information is composed of:

[0034] Then, the basis function H learned from the neural network １ , ..., H Ｎ depends on the viewing angle. １ , ..., k Ｎ The weights are changed and linearly combined. This determines the RGB values ​​at the aforementioned point when viewed from a certain angle. This type of modeling makes it possible to reproduce the phenomenon where the same location on an object looks different depending on the angle.

[0035] However, when using NeX, it is necessary to specify the resolution, position, and size of the MPIs to be placed in three-dimensional space before starting training. Here, if the MPI resolution (i.e., the amount of data) is fixed to a constant value, there is a trade-off between the resolution of the NVS image and the size of the range in which the radiance field is reconstructed.

[0036] In other words, in order to improve the perceived resolution of NVS images within the realistic constraints of data volume, it is necessary to reduce the size of the MPI and limit the range in which the radiance field is reconstructed. In this case, the presence of objects outside the range of the viewing frustum in which the MPI is placed becomes a problem.

[0037] In other words, when the input image for training contains a ray that observes a point on an object that exists outside the range of this viewing frustum, in order to reproduce the brightness value observed at this point, training is performed to reproduce that brightness value using other sampling points on the same ray that are included within the range of the viewing frustum.

[0038] This will be described in detail with reference to Fig. 5. As shown in Fig. 5, the MPI is placed based on the base camera viewpoint from the base camera, which is one of the cameras 50. The range of the view frustum in which this MPI is placed is the range in which the radiance field is reconstructed after learning the radiance field.

[0039] As explained with reference to Figure 2, the radiance field is learned by randomly selecting rays (pixels) from the input multi-viewpoint image and calculating the luminance values ​​corresponding to those rays by volume rendering.

[0040] That is, as shown in Fig. 5, a ray r0 from a camera viewpoint other than the base camera viewpoint is also selected. However, as shown by sampling point SP1 in the figure, if this ray r0 observes a point on an object that exists outside the range of the viewing frustum, in order to reproduce the luminance value observed at this point, learning is performed so that it is reproduced using at least one of the other sampling points on the same ray r0 within the range of the viewing frustum. This could result in an object surface being formed at a position where no object surface actually exists, which could cause the entire learning of the radiance field to fail.

[0041] Returning to the explanation of Fig. 1, in the information processing method according to this embodiment, as shown in Fig. 1, in learning a radiance field, light rays that observe an object that exists outside the learning range (i.e., the range of the MPI viewing frustum) are detected (step S1), and the detected light rays are then excluded from the learning target (step S2).

[0042] That is, the information processing method according to this embodiment adds a process of selecting light rays to be used for learning. Specifically, the information processing device 100 determines whether a light ray randomly selected from a multi-viewpoint image is a light ray that may be used for learning.

[0043] That is, the information processing device 100 determines whether the object observed by the light ray is within the range of the viewing frustum in which the MPI is placed and whether the light ray is, in principle, representable by the MPI. The information processing device 100 then uses representable light rays as learning objects and excludes unrepresentable light rays from the learning objects. That is, the information processing device 100 determines whether the light ray can be used in the learning process by determining whether the light ray can be represented by the MPI.

[0044] By performing such processing, it is possible to avoid the problem of learning failing due to the difference between the range of the viewing frustum in which the MPI is placed and the range in which the object actually exists, as shown in Figure 5.

[0045] An example of the configuration of the information processing device 100 to which the information processing method according to this embodiment is applied will be described in more detail below.

[0046] <<2. Configuration Example of Information Processing Device According to Embodiment of the Present Disclosure>> <2-1. Block Diagram> Fig. 6 is a block diagram showing a configuration example of an information processing device 100 according to an embodiment of the present disclosure. Note that in Fig. 6, only components necessary for explaining this embodiment are shown in functional blocks, and descriptions of general components are omitted.

[0047] In addition, in the description using FIG. 6, the description of components that have already been described may be simplified or omitted.

[0048] As shown in FIG. 6, the information processing device 100 includes an input unit 101 , an output unit 102 , a communication unit 103 , a storage unit 104 , and a control unit 105 .

[0049] The input unit 101 is a component that inputs data to the information processing device 100. The input unit 101 is realized by operation components such as a keyboard and a mouse, and accepts data input by a user.

[0050] The input unit 101 is realized as an input interface from various devices, for example, and accepts input of multi-viewpoint images captured by an RGB camera such as a smartphone or a single-lens camera.

[0051] The input unit 101 is realized by various sensors such as a depth sensor, a stereo camera, and a LiDAR (Light Detection and Ranging), and receives inputs such as depth measured by these various sensors.

[0052] The output unit 102 is a component that outputs data from the information processing device 100. The output unit 102 is realized by, for example, a head-mounted display or a stereoscopic display (a so-called "light field display"), and displays various information related to the learning results, such as NVS images rendered using the learning results.

[0053] The communication unit 103 is realized by, for example, a network adapter, and communicates with the external device 200. The external device 200 is a device, such as a game engine, that performs physical calculations based on data transmitted from the information processing device 100. The communication unit 103 transmits, for example, various pieces of information related to the learning results described above to the external device 200.

[0054] The storage unit 104 is realized by, for example, a semiconductor memory element such as a random access memory (RAM), a read only memory (ROM), or a flash memory, or a storage device such as a hard disk or an optical disk.

[0055] 6, the storage unit 104 stores radiance field information 104a. The radiance field information 104a is a training result of the radiance field, i.e., a trained model that has been trained to estimate an NVS image seen in a viewing direction from an arbitrary viewpoint position when the viewing direction is input. The radiance field information 104a includes the above-mentioned MPI.

[0056] The control unit 105 is a so-called controller, and is realized by, for example, a GPU (Graphics Processing Unit), a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc., executing a program according to this embodiment (not shown) stored in the storage unit 104 using RAM as a work area. The control unit 105 can also be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0057] The control unit 105 has an acquisition unit 105a, a learning unit 105b, a ray determination unit 105c, and a rendering unit 105d, and realizes or executes the information processing functions and actions described below while appropriately using the memory unit 104.

[0058] The acquisition unit 105a acquires multi-viewpoint images used for learning the radiance field via the input unit 101. The acquisition unit 105a also acquires camera parameters of each camera 50 using technology such as SfM. The acquisition unit 105a also acquires depths and the like measured by the various sensors described above via the input unit 101.

[0059] The learning unit 105b executes a learning process to learn a radiance field for generating a new viewpoint image according to an arbitrary viewing direction based on a multi-viewpoint image including a plurality of two-dimensional images. The ray determination unit 105c determines whether a ray of interest in the learning process can be used in the learning process based on the positional relationship between the ray of interest in the learning process and a predetermined spatial learning range of the radiance field.

[0060] Specifically, the learning unit 105b executes a learning process of a radiance field using MPI based on the multi-view images and camera parameters acquired by the acquisition unit 105a. In this learning process, the learning unit 105b randomly selects rays (pixels) from the multi-view images, and excludes from the learning target any rays that are determined by the ray determination unit 105c to be unable to be expressed in MPI (i.e., unable to be used in the learning process).

[0061] The learning unit 105b also learns rays that are determined by the ray determination unit 105c to be expressible in MPI (i.e., usable in the learning process). The learning unit 105b also saves the learning results in the radiance field information 104a.

[0062] Furthermore, the light ray determination unit 105c executes a light ray determination process, which will be described later. In the light ray determination process, it is determined whether a light ray randomly selected by the learning unit 105b is a light ray that can be expressed by MPI. That is, in the light ray determination process, light rays that observe an object that exists outside the range in which learning is performed, as shown in step S1, are detected. Details of the light ray determination process will be described later using FIGS. 8 and 9.

[0063] The rendering unit 105d generates an NVS image by performing volume rendering using the radiance field information 104a in accordance with, for example, a new viewing direction input via the input unit 101. The rendering unit 105d also outputs the generated NVS image to, for example, the output unit 102. The rendering unit 105d also calculates, for example, a luminance value corresponding to a ray during ray determination processing by volume rendering using a provisional MPI value during learning.

[0064] 2-2. Information Processing Procedure According to an Embodiment of the Present Disclosure> Next, an information processing procedure according to an embodiment of the present disclosure will be described with reference to FIGS. 7 to 9. FIG. 7 is a flowchart showing a processing procedure executed by an information processing device according to an embodiment of the present disclosure. FIG. 8 is an explanatory diagram of a light ray determination process. FIG. 9 is a flowchart showing the processing procedure of the light ray determination process.

[0065] Note that Fig. 7 corresponds to Fig. 2 already shown, and steps S101 to S106 in Fig. 2 correspond to steps S201, S202, S205, S206, S207, and S208 in Fig. 7, respectively. That is, Fig. 7 differs from Fig. 2 in that steps S203 and S204 are executed between steps S102 and S103 shown in Fig. 2.

[0066] As shown in FIG. 7 , in the information processing according to this embodiment, the control unit 105 acquires the captured multi-viewpoint images and the inside / outside camera parameters of each camera 50 estimated using a technique such as SfM (step S201), and starts the learning process of the radiance field using these as inputs.

[0067] In the learning process, the control unit 105 randomly selects a ray (pixel) from the acquired multi-viewpoint image (step S202), and then executes a ray determination process to determine whether the selected ray can be expressed in MPI (step S203).

[0068] In the ray determination process, as already described, the control unit 105 detects rays that observe an object that exists outside the range where learning is performed. First, consider the case where the depth of a randomly selected ray is obtained (known) before learning of the radiance field begins.

[0069] In this case, the control unit 105 determines the intersection point between the view frustum in which the MPI is placed and the selected ray, and determines the anteroposterior relationship between the intersection point and the pre-acquired depth of the ray. Specifically, as shown in Figure 8, if the intersection point P1 between the selected ray r1 and the view frustum is located in front of the known depth D1 of the pre-acquired ray r1 as viewed from the camera 50 that captured the image, the control unit 105 uses the ray r1 as a learning target for the radiance field.

[0070] This is because ray r1 is a ray that observes an object that exists inside the viewing frustum in which the MPI is placed, and so even if this ray r1 is used for learning, the learning breakdown described above does not occur.

[0071] On the other hand, as shown in Figure 8, if the intersection point P2 between the selected ray r2 and the viewing frustum is located further back from the camera 50 that took the photograph than the known depth D2 of the ray r2 that was previously acquired, this ray r2 is excluded from the learning target of the radiance field.

[0072] This makes it impossible to reconstruct the radiance field for this ray r2, but since ray r2 is a ray that observes an object that exists outside the viewing frustum in which the MPI is placed, it is possible to avoid learning failures that may occur when this ray r2 is used for learning.

[0073] In addition, methods for obtaining the depth of each ray include directly measuring the scene using the depth sensor mentioned above, using a point cloud obtained by SfM, or using a depth map obtained by multi-view stereo.

[0074] Alternatively, a depth map estimated by a learning-based monocular depth estimator can be used. In this case, the scale of the estimated depth map is indefinite, but a scale-corrected depth map can be obtained by aligning it with a point cloud obtained by SfM or the like. The scale can be corrected using, for example, the least squares method.

[0075] Next, consider the case where the depth corresponding to a randomly selected ray is not obtained in advance (is unknown). In this case, the control unit 105 calculates the intersection point between the view frustum in which the MPI is located and the selected ray.

[0076] Then, when a ray intersects with both the near clip plane and the far clip plane of the view frustum in which the MPI is located, the control unit 105 uses this ray as a learning target. In the example of Figure 8, the control unit 105 uses ray r3 as a learning target because it intersects with both the near clip plane NP and the far clip plane FP. This is because ray r3 can be represented in any layer of the MPI regardless of the depth at which it observes an object.

[0077] On the other hand, if the selected ray does not intersect with both the near clip plane NP and the far clip plane FP of the viewing frustum in which the MPI is placed, such as ray r4 in Figure 8, the control unit 105 determines whether this ray observes an object inside the viewing frustum in which the MPI is placed.

[0078] Specifically, if the selected ray does not intersect with both the near clip plane NP and the far clip plane FP, the control unit 105 makes this determination based on the error between the generated image generated using the latest MPI and the input image.

[0079] More specifically, the control unit 105 calculates the brightness value corresponding to the ray of interest by volume rendering using provisional MPI values ​​during learning, and determines the error between the calculated brightness value and the true brightness value of the input image. If the error is equal to or less than a predetermined threshold, the control unit 105 determines that the ray observed an object present inside the viewing frustum in which the MPI is located, and uses this ray as a learning target.

[0080] In contrast, if the error exceeds a predetermined threshold, the control unit 105 determines that the ray observed an object outside the viewing frustum in which the MPI is placed, and excludes the ray from the learning target.

[0081] By using this type of ray determination processing, the control unit 105 can detect rays that are not necessary for learning, even if they are difficult to determine based solely on geometric information based on the intersection of the ray and the viewing frustum.

[0082] The processing procedure of the above-mentioned light ray determination process is shown in Fig. 9. As shown in Fig. 9, in the light ray determination process, the control unit 105 calculates the intersection point between the selected light ray and the view frustum (step S301). Then, the control unit 105 determines whether the depth of the light ray has been acquired (step S302).

[0083] If the depth of the ray has already been acquired (step S302, Yes), the control unit 105 determines whether the intersection point is in front of the depth (step S303). If it is in front (step S303, Yes), the control unit 105 determines that the ray can be expressed by MPI (step S304). On the other hand, if it is not in front (step S303, No), the control unit 105 determines that the ray cannot be expressed by MPI (step S305).

[0084] If the depth of the ray has not been acquired (step S302, No), the control unit 105 determines whether the ray has intersections with both the near clip plane NP and the far clip plane FP (step S306). If the ray has intersections with both clip planes (step S306, Yes), the control unit 105 determines that the ray can be expressed in MPI (step S307).

[0085] On the other hand, if the ray does not intersect with both clipping planes (step S306, No), the control unit 105 calculates the luminance value corresponding to the ray of interest by volume rendering using a provisional MPI value during learning (step S308).The control unit 105 then determines whether the error between the calculated luminance value and the luminance value of the input image is equal to or less than a predetermined threshold (step S309).

[0086] If the error is equal to or smaller than the threshold (step S309, Yes), the control unit 105 determines that the ray can be expressed by MPI (step S307).If the error exceeds the threshold (step S309, No), the control unit 105 determines that the ray cannot be expressed by MPI (step S310).

[0087] This ray determination process calculates the brightness error for each ray individually and then performs a determination based on a threshold value. However, under the assumption that the rays to be excluded from the learning target exist spatially continuously, information on rays near the ray of interest may also be used in order to improve the accuracy of determining whether learning is possible.

[0088] The term "neighboring rays" used here refers to rays (pixels) adjacent to the ray (pixel) of interest in each image plane of the input image. In this case, four, eight, or more neighboring rays (pixels) may be used. Specifically, the following methods can be considered.

[0089] First, the control unit 105 calculates the luminance values ​​of the ray of interest and its neighboring rays using provisional MPI values ​​obtained during learning. Next, the control unit 105 generates a residual image by calculating the error between these luminance values ​​and the luminance values ​​of the input image. The control unit 105 then performs filtering such as morphological transformation on this residual image. The control unit 105 then performs a judgment on the filtered results using the aforementioned threshold. By performing such processing, it is expected that the judgment accuracy will be improved compared to when judgment is performed individually for each ray.

[0090] Furthermore, in the ray determination process, image feature error may be used instead of brightness value error in order to improve robustness against changes in the shooting conditions of the input image, changes in lighting, etc. Image feature extraction can use trained neural networks proposed in, for example, the non-patent literature "Caron, Mathilde, et al. "Emerging properties in self-supervised vision transformers." Proceedings of the IEEE / CVF international conference on computer vision. 2021." and the non-patent literature "Li, Boyi, et al. "Language-driven semantic segmentation." arXiv preprint arXiv:2201.03546 (2022)."

[0091] Returning to the description of Fig. 7, after performing the ray determination process in step S203, the control unit 105 determines whether the selected ray can be expressed in MPI based on the determination result of the ray determination process (step S204).

[0092] If the ray can be expressed by MPI (Yes at step S204), the control unit 105 calculates the brightness value corresponding to the ray by volume rendering using the latest MPI value (step S205).

[0093] Then, based on the error between the brightness value estimated by volume rendering and the brightness value of the input image, the control unit 105 updates the radiance field information (i.e., MPI) by the backpropagation algorithm (step S206). If the ray cannot be expressed by MPI (step S204, No), the control unit 105 repeats the learning process from step S202.

[0094] The control unit 105 then performs the above MPI update process a sufficient number of times and determines whether the learning has converged based on the number of times (step S207). If the learning has not converged (step S207, No), the control unit 105 repeats the learning process from step S202. If the learning has converged (step S207, Yes), the control unit 105 ends the learning process and saves the MPI (step S208).

[0095] <<3. Modifications>> Although the embodiment of the present disclosure has been described, the information processing method according to the present embodiment can be modified in several other ways.

[0096] For example, although the above description has been given using radiance field learning using MPI as an example, scene representation methods other than MPI may also be used. Examples of scene representation methods other than MPI include voxel grids, tensor decomposition, and hash tables. These scene representation methods, like MPI, require a spatial learning range for reconstructing the radiance field to be set in advance before learning begins. Therefore, if light rays observed for an object outside the learning range are used in learning, learning may fail. To address this issue, the same effect as in this embodiment can be achieved by performing the ray determination process based on the intersection of light rays and the learning range proposed in this embodiment.

[0097] Up to this point, the spatial range of the view frustum set in advance in learning the radiance field has been assumed to be fixed. However, if the ray of interest is determined by the ray determination process shown in Figures 8 and 9 to be a ray that cannot be represented by MPI, i.e., a ray that cannot be used for learning, a step of expanding the view frustum to the required range so that it intersects with the ray of interest may be included. This step, at the expense of an increase in the total amount of MPI data, makes it possible to avoid learning failures and reconstruct the radiance field of the object observed by the ray of interest.

[0098] Furthermore, among the processes described in the above-mentioned embodiments of the present disclosure, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods.

[0099] 8 and 9 automatically determines whether a ray can be used for learning by using one or both of the geometric positional relationship between the ray and the learning range of the radiance field and the error between the input image and the generated image generated from the MPI during learning. However, part of this ray determination process may be performed by the user. For example, before starting learning, the user may specify some rays in the input image as not to be learned via the input unit 101 or the like.

[0100] In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0101] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0102] The above-described embodiments of the present disclosure can be combined as appropriate within the scope of the present disclosure without causing any inconsistency in the processing content. The order of the steps shown in the sequence diagrams or flowcharts of the present embodiments can be changed as appropriate.

[0103] <<4. Hardware Configuration>> The information processing device 100 according to the embodiment of the present disclosure described above is realized by a computer 1000 configured as shown in Fig. 10, for example. Fig. 10 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing device 100. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, a secondary storage device 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0104] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the secondary storage device 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the secondary storage device 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0105] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0106] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records the program according to this embodiment, which is an example of the program data 1450.

[0107] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550. For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0108] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0109] For example, when the computer 1000 functions as the information processing device 100, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to realize the functions of the control unit 105. The secondary storage device 1400 also stores the program according to this embodiment, various models used in the information processing according to this embodiment, and various data as appropriate. The CPU 1100 reads and executes program data 1450 from the secondary storage device 1400, but as another example, the CPU 1100 may also acquire these programs from other devices via an external network 1550.

[0110] <<5. Conclusion>> As described above, according to one embodiment of the present disclosure, the information processing device 100 includes a learning unit 105b and a ray determination unit 105c. The learning unit 105b executes a learning process to learn a radiance field for generating a new viewpoint image according to an arbitrary viewing direction, based on a multi-viewpoint image including a plurality of two-dimensional images. The ray determination unit 105c determines whether a ray of interest in the learning process can be used in the learning process, based on the positional relationship between the ray and a predetermined spatial learning range of the radiance field. This makes it possible to avoid failure of learning of the radiance field in learning-based NVS.

[0111] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0112] Furthermore, the effects of each embodiment described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0113] Note that the present technology may also be configured as follows. (1) An information processing device comprising: a learning unit that executes a learning process to learn a radiance field for generating a new viewpoint image according to an arbitrary viewing direction, based on a multi-viewpoint image including a plurality of two-dimensional images; and a ray determination unit that determines whether a ray of interest in the learning process can be used in the learning process, based on a positional relationship between a ray of interest in the learning process and a predetermined spatial learning range of the radiance field. (2) The information processing device described in (1), wherein, when the depth of the ray is known, the ray determination unit determines whether the ray can be used based on a positional relationship between an intersection point between the ray and the learning range and the depth. (3) The information processing device described in (2), wherein the ray determination unit determines that the ray can be used in the learning process if the intersection point is closer to the depth as viewed from a camera viewpoint observing the ray, and determines that the ray cannot be used in the learning process if the intersection point is deeper than the depth. (4) The information processing device according to (1), wherein the learning unit sets the learning range as a view frustum based on an arbitrary camera viewpoint in the multi-view image, and the ray determination unit, when the depth of the ray is unknown, determines whether the ray can be used based on a positional relationship between the ray and a clip plane in the view frustum. (5) The information processing device according to (4), wherein the ray determination unit determines that the ray can be used in the learning process when the ray has an intersection with both a near clip plane and a far clip plane in the view frustum. (6) The information processing device according to (5), wherein, when the ray does not have an intersection with both the near clip plane and the far clip plane, the ray determination unit determines whether the ray can be used based on an error between an image generated using the radiance field, including a case in which learning is in progress, and an input image in which the ray is observed. (7) The information processing device according to (6), wherein the ray determination unit determines that the ray can be used in the learning process when the error is equal to or smaller than a predetermined threshold, and determines that the ray cannot be used in the learning process when the error exceeds the threshold.(8) The information processing device according to any one of (4) to (7), wherein the learning unit performs the learning process to learn the radiance field using multiplane images (MPIs) arranged in the viewing frustum. (9) An information processing method comprising: performing a learning process to learn a radiance field for generating a new viewpoint image according to an arbitrary viewing direction, based on multi-viewpoint images including a plurality of two-dimensional images, and determining whether the light ray can be used in the learning process based on a positional relationship between a light ray of interest in the learning process and a predetermined spatial learning range of the radiance field. (10) A computer-readable non-transitory storage medium storing a program for causing a computer to perform the steps of: performing a learning process to learn a radiance field for generating a new viewpoint image according to an arbitrary viewing direction, based on multi-viewpoint images including a plurality of two-dimensional images, and determining whether the light ray can be used in the learning process based on a positional relationship between a light ray of interest in the learning process and a predetermined spatial learning range of the radiance field.

[0114] 50 Camera 100 Information processing device 101 Input unit 102 Output unit 103 Communication unit 104 Storage unit 104a Radiance field information 105 Control unit 105a Acquisition unit 105b Learning unit 105c Light ray determination unit 105d Rendering unit 200 External device

Claims

1. An information processing device comprising: a learning unit that executes a learning process to learn a radiance field for generating a new viewpoint image according to an arbitrary viewing direction based on a multi-viewpoint image including a plurality of two-dimensional images; and a light ray determination unit that determines whether or not a light ray can be used in the learning process based on the positional relationship between a light ray focused on in the learning process and a predetermined spatial learning range of the radiance field.

2. The information processing device according to claim 1, wherein, when the depth of the ray is known, the light ray determination unit determines whether or not the ray can be used based on the positional relationship of the intersection point between the ray and the learning range and the depth.

3. The information processing device of claim 2, wherein the light ray determination unit determines that the light ray can be used in the learning process if the intersection point is closer than the depth when viewed from the camera viewpoint observing the light ray, and determines that the light ray cannot be used in the learning process if the intersection point is deeper than the depth.

4. The information processing device described in claim 1, wherein the learning unit sets the learning range as a view frustum based on any camera viewpoint in the multi-view image, and the ray determination unit, when the depth of the ray is unknown, determines whether the ray can be used based on the positional relationship between the clip plane in the view frustum and the ray.

5. The information processing device according to claim 4, wherein the ray determination unit determines that the ray can be used in the learning process when the ray has an intersection with both a near clip plane and a far clip plane in the view frustum.

6. The information processing device described in claim 5, wherein the ray determination unit determines whether or not the ray can be used based on the error between an image generated using the radiance field, including a case in which learning is in progress, and an input image in which the ray is observed, when the ray does not have an intersection with both the near clip plane and the far clip plane.

7. The information processing device according to claim 6, wherein the light ray determination unit determines that the light ray can be used in the learning process if the error is equal to or less than a predetermined threshold, and determines that the light ray cannot be used in the learning process if the error exceeds the threshold.

8. The information processing device according to claim 4, wherein the learning unit executes the learning process of learning the radiance field using multiplane images (MPIs) arranged in the view frustum.

9. An information processing method comprising: executing a learning process to learn a radiance field for generating a new viewpoint image according to an arbitrary viewing direction based on a multi-view image including a plurality of two-dimensional images; and determining whether or not a ray of interest in the learning process can be used in the learning process based on the positional relationship between the ray of interest in the learning process and a predetermined spatial learning range of the radiance field.

10. A computer-readable non-transitory storage medium storing a program that causes a computer to execute a learning process to learn a radiance field for generating a new viewpoint image according to an arbitrary viewing direction based on a multi-viewpoint image including multiple two-dimensional images, and determine whether or not a light ray focused on in the learning process can be used in the learning process based on the positional relationship between the light ray and a predetermined spatial learning range of the radiance field.

Images