Information processing device, information processing method, and computer-readable non-transitory storage medium

Abstract

This information processing device comprises an estimation unit and a super-resolution processing unit. The estimation unit estimates a new viewpoint image, which corresponds to a discretionary line-of-sight direction, by using a first model trained so as to estimate the new viewpoint image on the basis of a multi-viewpoint image including a plurality of two-dimensional images. The super-resolution processing unit performs super-resolution processing on the estimated new viewpoint image. The estimation unit estimates a processing target region, in which the super-resolution processing is performed, and a non-processing target region, in which the super-resolution processing is not performed, by using a second model trained so as to estimate the processing target region and the non-processing target region for each new viewpoint image in accordance with the line-of-sight direction. The super-resolution processing unit performs the super-resolution processing on the estimated processing target region and does not perform the super-resolution processing on the estimated non-processing target region.

Description

Information processing device, information processing method, and computer-readable non-transitory storage medium

[0001] The present disclosure relates to an information processing device, an information processing method, and a computer-readable non-transitory storage medium.

[0002] In recent years, a technology called Neural Radiance Fields (NeRF) has been proposed and developed, which can reproduce any novel viewpoint image (Novel View or Novel View Image) using input images from multiple viewpoints and camera positions and orientations (see, for example, Non-Patent Document 1).

[0003] Initially, NeRF required several tens of seconds for novel view synthesis, making it difficult to perform real-time rendering for a new input gaze direction, but recently, several technologies have been proposed to improve real-time performance, for example, to around 60 fps in Full HD (see, for example, Non-Patent Document 2). However, NeRF has the problem that the output image is blurred compared to the input image, making it difficult to reproduce a sharp image like a real photo.

[0004] To address this issue, one signal processing solution is to apply learning-based super-resolution (SR) technology to the output image of NeRF, enabling image restoration with sharpness closer to that of the input image. On the other hand, a method has also been proposed that combines "Direct Voxel Grid Optimization," a NeRF technique, with learning-based super-resolution processing, and has been shown to produce output images that are sharper than those obtained with conventional NeRF (see, for example, Non-Patent Document 3).

[0005] However, super-resolution processing also takes time, making it difficult to achieve real-time rendering at high resolutions (for example, 4K or 8K at 60 fps or higher) when combined with NeRF.

[0006] Here, as a method for speeding up super-resolution processing, a method has been proposed in which the complexity of each image patch, which is a predetermined pixel unit, is determined, and then switching between lightweight super-resolution processing and heavy super-resolution processing for each image patch according to the determination result (see, for example, non-patent document 4).

[0007] However, since determining the complexity of each image patch also takes time, it is difficult to achieve real-time rendering with this method. One possible solution is to perform the determination process in advance and save the results, but in the case of new viewpoint images, saving the determination results for each viewpoint to be output is not realistic in terms of data volume, although this may be possible for two-dimensional images.

[0008] As another prior art example of switching super-resolution processing for each image patch, a technique has been proposed in which super-resolution processing is switched depending on the accuracy of motion vectors of a moving image (see, for example, Patent Document 1).

[0009] Mildenhall, Ben, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.” arXiv [cs.CV]. arXiv. http: / / arxiv.org / abs / 2003.08934.Wizadwongsa, Suttisak, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn. 2021. “NeX: Real-Time View Synthesis with Neural Basis Expansion.” arXiv [cs.CV]. arXiv. http: / / arxiv.org / abs / 2103.05606.Wang, Zhongshu, Lingzhi Li, Zhen Shen, Li Shen, and Liefeng Bo. 2022. “4K-NeRF: High Fidelity Neural Radiance Fields at Ultra High Resolutions.” arXiv [cs.CV]. arXiv. http: / / arxiv.org / abs / 2212.04701.Kong, Xiangtao, Hengyuan Zhao, Yu Qiao, and Chao Dong. 2021. “ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic.” https: / / openaccess.thecvf.com / content / CVPR2021 / papers / Kong_ClassSR_A_General_Framework_to_Accelerate_Super-Resolution_Networks_by_Data_CVPR_2021_paper.pdf.

[0010] Japanese Patent Application Laid-Open No. 2018-023034

[0011] However, although the above-mentioned conventional technology can speed up super-resolution processing by switching super-resolution processing for each image patch depending on the accuracy of motion prediction, it is a technology that improves the image quality of two-dimensional video using codec information of the two-dimensional video, and therefore cannot be directly applied when performing super-resolution processing on the rendering results of a new viewpoint image.

[0012] Therefore, the present disclosure proposes an information processing device, an information processing method, and a computer-readable non-transitory storage medium that are capable of speeding up super-resolution processing when super-resolution processing is performed on the rendering results of a new viewpoint image.

[0013] In order to solve the above-described problems, an information processing device according to one embodiment of the present disclosure includes an estimation unit and a super-resolution processing unit. The estimation unit estimates a new viewpoint image using a first model trained to estimate the new viewpoint image according to an arbitrary line of sight direction based on a multi-viewpoint image including a plurality of two-dimensional images. The super-resolution processing unit performs super-resolution processing on the estimated new viewpoint image. The estimation unit also estimates a processing target region to be subjected to the super-resolution processing and a non-processing target region to be not subjected to the super-resolution processing for each new viewpoint image using a second model trained to estimate the processing target region to be subjected to the super-resolution processing and a non-processing target region to be not subjected to the super-resolution processing according to the line of sight. The super-resolution processing unit also performs the super-resolution processing on the estimated processing target region and does not perform the super-resolution processing on the estimated non-processing target region.

[0014] FIG. 4 is a diagram illustrating a basic processing flow before super-resolution processing according to an embodiment of the present disclosure. FIG. 5 is an image diagram of MPI. FIG. 6 is a conceptual diagram of formula (b). FIG. 7 is a supplementary explanatory diagram for FIG. 3. FIG. 8 is an image diagram of an information processing method according to the present embodiment. FIG. 9 is a diagram illustrating an example of an NeRF output image, an image after super-resolution processing, and a difference image. FIG. 10 is a diagram illustrating an overview of an information processing method according to the present embodiment. FIG. 11 is a block diagram illustrating an example configuration of an information processing device according to an embodiment of the present disclosure. FIG. 12 is a flowchart illustrating a processing procedure for learning a difference map model for generating a switching map executed by an information processing device. FIG. 13 is a diagram illustrating parameters added in the learning processing of the difference map model. FIG. 14 is a flowchart illustrating a processing procedure for real-time rendering processing using a switching map executed by an information processing device. FIG. 15 is a diagram illustrating an example of a difference map. FIG. 16 is a diagram illustrating an example of a super-resolution processing switching map. FIG. 17 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of an information processing device.

[0015] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0016] In the following, it is assumed that an information processing device according to an embodiment of the present disclosure (hereinafter referred to as "the present embodiment") is an information processing device 100 (see FIG. 8) that renders a new viewpoint image according to the line of sight direction using NeRF, performs super-resolution processing on the output image after rendering (hereinafter referred to as "NeRF output image"), and outputs a super-resolution processed image. In the following, it is assumed that an information processing method according to an embodiment of the present disclosure is an information processing method executed by the information processing device 100.

[0017] The present disclosure will be described in the following order: 1. Overview 1-1. Description of NeRF 1-2. Description of NeX rendering process 1-3. Description of super-resolution process and switching process therefor 2. Configuration example of information processing device 2-1. Block diagram 2-2. Processing procedure for learning process of difference map model for generating switching map 2-3. Processing procedure for real-time rendering process using switching map 3. Modified example 4. Hardware configuration 5. Conclusion

[0018] <<1. Overview>> <1-1. Description of NeRF> Fig. 1 is a diagram showing the flow of basic processing before super-resolution processing in an embodiment of the present disclosure. "NeRF" is a technology that generates a new viewpoint image viewed from an arbitrary new viewpoint in a three-dimensional space in which a subject object exists, by deep learning using multi-viewpoint images, which are a group of two-dimensional images captured from various angles by a camera 50, as shown in Fig. 1.

[0019] As shown in Fig. 1, in the information processing method according to this embodiment, deep learning for NeRF (hereinafter referred to as "NeRF learning") is performed using multi-viewpoint images as a data set to generate an NeRF model 104a. The NeRF model 104a is a neural network that is trained to estimate a new viewpoint image seen from an arbitrary viewpoint position when the direction of gaze from the viewpoint is input. In other words, the NeRF model 104a is a model for synthesizing a radiance field using a neural network.

[0020] Next, in the information processing method according to this embodiment, when an arbitrary gaze direction is input, a new viewpoint image seen from the gaze direction is estimated using the NeRF model 104a (hereinafter referred to as "NeRF estimation" as appropriate), and an NeRF output image is generated as the NeRF estimation result, i.e., the rendering result. The arrows in the NeRF output image shown in Figure 1 schematically represent changes in the position and orientation of an object (here, a cat) according to changes in the viewpoint position and the gaze direction.

[0021] In the information processing method according to this embodiment, super-resolution processing is performed on the NeRF output image, and the image after super-resolution processing is displayed on a display device such as a display.

[0022] Photogrammetry, a similar model-based technique, has existed for some time, but it was difficult to reproduce information such as the degree of light reflection and the appearance of scenery seen through a screen. However, "NeRF" uses a neural network to accurately estimate the degree to which an object exists at each point and its color distribution (Radiance Fields, hereafter referred to as "radiance fields") from information on the position and viewing angle in three-dimensional space. Converting the radiance fields into two-dimensional images using a rendering method called volume rendering makes it possible to express new viewpoint images.

[0023] <1-2. Explanation of NeX Rendering Process> While NeRF has traditionally been capable of rendering high-resolution new viewpoint images, volume rendering using a neural network has had the problem of requiring a long processing time. Specifically, it takes about 30 seconds per viewpoint even when processing using a GPU (Graphical Processing Unit).

[0024] In response to this, many techniques have been proposed recently to shorten rendering times by caching neural network calculation results in various formats. In a technique called "NeX" disclosed in Non-Patent Document 2, intermediate data of the NeRF model 104a is cached in the format of multiplane images (MPI) in which multiple layers are arranged in the depth direction, thereby speeding up volume rendering processing by approximately 1000 times compared to NeRF, making real-time rendering possible.

[0025] FIG. 2 is a conceptual diagram of an MPI. As shown in FIG. 2, an MPI typically has a structure consisting of a group of RGBA layers in which semi-transparent RGBA images are stacked in a frustum shape in three-dimensional space to cover the entire scene, from the object closest to the camera 50 that captured the scene to a sufficiently distant object. A new viewpoint image is generated by overlaying each RGBA image that constitutes the MPI while shifting them according to the line of sight from each viewpoint. Note that the first viewpoint in FIG. 2 is, for example, the shooting position of the camera 50. The second viewpoint is, for example, a new arbitrary viewpoint position that is not the shooting position of the camera 50.

[0026] The radiance field in NeX is trained so that the RGB values ​​obtained by volume rendering these MPIs are optimized to approximate the RGB values ​​captured by the camera 50.

[0027] In the rendering process of NeX, rays are traced from the line of sight, and the existence probability α at the point where the ray intersects with the MPI is multiplied by the color of that point, and then added. The following formula (a) expresses the existence probability A (= {α １ , α ２ , …, α Ｄ}) and color component C (= {c １ , c ２ , ..., c Ｄ}) is an equation representing a method for calculating the RGB values ​​of an output image when

[0028]

[0029] c in this formula (a) ｄ is the angle-independent color component k ０ and changes depending on the viewing angle (View Dependent, hereinafter referred to as "VD"), that is, the angle-dependent color component k １ ~k N The following formula (b) is calculated by weighting and adding c ｄ This is the formula for calculating

[0030]

[0031] 3 is a conceptual diagram of formula (b). Also, FIG. 4 is a supplementary explanatory diagram of FIG. 3. As shown in FIG. 3, any one point (here, the point in the upper right corner) in a certain layer in the NeRF model 104a of NeX is calculated by the above k ０ and this k ０ k when viewed from various angles １ , ..., k Ｎ The reflection coefficient is:

[0032] Then, the basis function H learned from the neural network １ , ..., H Ｎ depends on the viewing angle. １ , ..., k Ｎ The weights are changed and linearly combined. This determines the RGB values ​​at the aforementioned point when viewed from a certain angle. This type of modeling makes it possible to reproduce the phenomenon where the same location on an object looks different depending on the angle.

[0033] Although there may be some overlapping explanations, the parameters shown in FIG. 3 will be supplemented with FIG. 4, where the aforementioned α is the existence probability in three-dimensional coordinates and corresponds to the "geometry component" (see "1) Alpha" in the figure). ０ is angle-independent color information, and corresponds to the "texture component" (see "2) Color" in the figure).

[0034] Also, k １ ~k ４ ... is angle-dependent color information, and corresponds to the "VD component" (see "3) Color_VD" in the figure). １ ~H ４ ... is angle-dependent color information and corresponds to a "basis function" (see "4) Basis" in the figure).

[0035] <1-3. Description of super-resolution processing and switching processing therefor> Next, super-resolution processing and switching processing therefor will be described with reference to Figs. 5 to 7. Fig. 5 is an image diagram of the information processing method according to this embodiment. Fig. 6 is a diagram showing an example of an NeRF output image, an image after super-resolution processing, and a difference image. Fig. 7 is a diagram outlining the information processing method according to this embodiment.

[0036] Super-resolution processing is a technology that obtains a high-resolution output image from a low-resolution input image. Among super-resolution processes, learning-based super-resolution processing can generate a high-resolution image from a single input image and is known as a method with high restoration accuracy. As shown in the "NeRF output image" and "image after super-resolution processing (100%)" in Figure 5, performing super-resolution processing on the NeRF output image can improve the perceived resolution of the image, for example, by sharpening the doll's hair in Figure 5.

[0037] However, since super-resolution processing also requires computational costs, it is necessary to speed up the super-resolution processing in order to achieve real-time rendering using NeRF. To obtain the "image after super-resolution processing (100%)" in Figure 5, it would take the processing time required to perform super-resolution processing on all pixels (i.e., 100%) of the NeRF output image.

[0038] The information processing method according to this embodiment is a study of super-resolution processing switching processing for speeding up this super-resolution processing. The super-resolution processing switching processing refers to a process of dividing an image into image patches of predetermined pixel units (e.g., 128 x 128 pixels) and switching whether or not to perform super-resolution processing for each image patch, as shown in the "switching map (50% ON)" in Fig. 5 . An image patch on which super-resolution processing is performed corresponds to a "processing target area." An image patch on which super-resolution processing is not performed corresponds to a "non-processing target area." Note that in this embodiment, when super-resolution processing is performed, it corresponds to "ON." When super-resolution processing is not performed, it corresponds to "OFF."

[0039] In the information processing method according to this embodiment, NeRF learning is performed on the difference image before and after super-resolution processing. FIG. 6 shows an example of an NeRF output image, an image after super-resolution processing, and a difference image. In the information processing method according to this embodiment, NeRF learning is performed on the difference image between the NeRF output image and the image after super-resolution processing for each viewpoint of multiple two-dimensional images included in the multi-viewpoint image, as shown in FIG. 6. Then, when a new viewpoint and line of sight direction are input, image patches that are estimated to have a large three-dimensional difference amount before and after super-resolution processing are set to ON, and image patches that are estimated to have a small difference amount are set to OFF.

[0040] For example, even if the image patch is turned ON so that super-resolution processing is performed on half of the total number of image patches (i.e., 50%), an output image can be generated that looks almost identical to the "image after super-resolution processing (100%)," as shown in "image after super-resolution processing (50%)" in Figure 5.

[0041] To achieve this processing, it is necessary to generate a super-resolution processing switching map (hereinafter referred to as "switching map" as appropriate) that enables highly accurate ON / OFF switching of super-resolution processing. The switching map corresponds to an example of "map information." In conventional technology, processing is switched using a network for determining complexity for each image patch, or processing is switched based on the reliability of the motion vector. In this embodiment, a method for generating a switching map according to three-dimensional representation using NeRF is shown.

[0042] Although an example using image patches has been described here, the predetermined pixel unit that is the unit for switching ON / OFF of the super-resolution processing may be every pixel. The unit for switching ON / OFF can be appropriately selected depending on, for example, the performance of the GPU included in the information processing device 100.

[0043] An overview of an information processing method according to this embodiment for generating a switching map will be described. As shown in FIG. 7 , in the information processing method according to this embodiment, a multi-viewpoint difference image is obtained, which is a difference image between an NeRF output image for each viewpoint of the above-mentioned plurality of two-dimensional images and an image after super-resolution processing. Then, NeRF learning is performed using this multi-viewpoint difference image as a data set to generate a difference map model 110a. The difference map model 110a is a neural network that is trained to estimate a difference image viewed from an arbitrary viewpoint position when a line of sight direction from this line of sight direction is input. In other words, the difference map model 110a is a model for synthesizing a radiance field using a neural network.

[0044] In this case, the difference map model 110a may be trained by transfer learning based on the NeRF model 104a. Since the NeRF model 104a has already learned the existence probability α of an object in three-dimensional space, the learning can be converged quickly by transfer learning. Details of this point will be described later using FIG. 10.

[0045] Next, in the information processing method according to this embodiment, when an arbitrary gaze direction is input, NeRF estimation is performed using the difference map model 110a, and a difference image viewed from the gaze direction is rendered and output as a difference map.

[0046] In the information processing method according to the present embodiment, the switching map is generated based on the difference map, for example, by reducing the resolution of the difference map for each of the image patches.

[0047] In the information processing method according to this embodiment, the aforementioned ON / OFF setting is performed for each image patch according to the generated switching map, and super-resolution processing is performed only on image patches that are set to ON.

[0048] <<2. Configuration Example of Information Processing Device>> <2-1. Block Diagram> Next, a configuration example of an information processing device 100 that executes the information processing method according to the present embodiment will be described. Fig. 8 is a block diagram showing a configuration example of the information processing device 100 according to an embodiment of the present disclosure. Note that Fig. 8 shows only the components necessary for explaining this embodiment in functional blocks, and a description of general components is omitted.

[0049] In addition, in the description using FIG. 8, the description of components that have already been described will be appropriately simplified or omitted.

[0050] As shown in Figure 8, the information processing device 100 has a camera position and orientation input unit 101, a multi-viewpoint image input unit 102, an NeRF learning unit 103, an NeRF model storage unit 104, an NeRF estimation unit 105, an NeRF output image storage unit 106, and a super-resolution processing unit 107.

[0051] The information processing device 100 also has a super-resolution processed image storage unit 108, a difference image calculation unit 109, a difference map model storage unit 110, a difference map storage unit 111, a switching map generation unit 112, a gaze direction input unit 113, and a switching threshold setting unit 114.

[0052] Although not shown in FIG. 8, the information processing device 100 may be connected to input devices such as a keyboard and a mouse that accept general input operations from a user, and an output device such as a display.

[0053] Although not shown, the information processing device 100 also includes a storage unit and a control unit. The storage unit is realized by a storage device such as a random access memory (RAM), a read only memory (ROM), a flash memory, or a hard disk drive (HDD).

[0054] The storage unit includes, for example, an NeRF model storage unit 104 , an NeRF output image storage unit 106 , a super-resolution processed image storage unit 108 , a difference map model storage unit 110 , and a difference map storage unit 111 .

[0055] The control unit controls each unit of the information processing device 100. The control unit is realized by, for example, a GPU, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc., executing a program according to the present embodiment (not shown) stored in the storage unit using RAM as a work area. The control unit can also be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0056] The control unit has, for example, a camera position and orientation input unit 101, a multi-viewpoint image input unit 102, an NeRF learning unit 103, an NeRF estimation unit 105, a super-resolution processing unit 107, a difference image calculation unit 109, a switching map generation unit 112, a gaze direction input unit 113, and a switching threshold setting unit 114, and realizes or executes the functions and actions of the information processing described below.

[0057] <2-2. Processing Procedure for Learning Process of Difference Map Model for Generating Switching Map> Next, the processing procedure for learning process of difference map model 110a for generating switching maps executed by information processing device 100 will be described using already shown Fig. 8 as well as Figs. 9 and 10. Fig. 9 is a flowchart showing the processing procedure for learning process of difference map model 110a for generating switching maps executed by information processing device 100. Fig. 10 is a diagram showing parameters added in the learning process of difference map model 110a.

[0058] The switching map according to this embodiment is required to be map information that can determine whether the difference between the input image and the output image of the super-resolution processing is large or small. Furthermore, it is desirable that this map information can represent values ​​that change according to viewpoint transformation, and can be calculated quickly with low calculation cost.

[0059] Therefore, in this embodiment, NeRF learns the difference between the rendering image using NeRF (i.e., the NeRF output image) and the super-resolution processed image, which is the result of performing super-resolution processing on the NeRF output image, and generates map information that enables three-dimensional real-time rendering of this difference.

[0060] 8 , in normal NeRF learning, the NeRF learning unit 103 learns the three-dimensional existence probability α and color information that changes depending on the viewpoint from the camera position and orientation and the corresponding multi-viewpoint image input from the camera position and orientation input unit 101 and the multi-viewpoint image input unit 102. Then, the NeRF learning unit 103 outputs the resulting NeRF model 104a to the NeRF model storage unit 104.

[0061] Furthermore, the NeRF estimation unit 105 estimates the viewpoint and the image seen from the line of sight direction input from the line of sight direction input unit 113 according to the NeRF model 104a stored in the NeRF model storage unit 104, and outputs the image as an NeRF output image to the NeRF output image storage unit 106.

[0062] Furthermore, if super-resolution processing is to be performed thereafter, the super-resolution processing unit 107 performs super-resolution processing on the NeRF output image stored in the NeRF output image storage unit 106, and outputs the image to the super-resolution processed image storage unit 108 as a super-resolution processed image.

[0063] In contrast to the normal NeRF learning and super-resolution processing described above, the learning process of the difference map model 110a for generating a switching map has the following procedure. As shown in Fig. 9 , first, the difference image calculation unit 109 calculates a difference image between an NeRF output image and a super-resolution processed image obtained by performing super-resolution processing on the NeRF output image (step S101). That is, the difference image calculation unit 109 calculates a difference image between a new viewpoint image for each viewpoint of multiple two-dimensional images included in the multi-viewpoint images used in the NeRF learning of the NeRF model 104a and the super-resolution processed image.

[0064] The NeRF output image is stored in the NeRF output image storage unit 106, and the super-resolution processed image is stored in the super-resolution processed image storage unit 108. As already shown in Fig. 6, it can be seen that the difference image calculated at this time tends to have a small difference amount in flat regions with few edges (see the dark parts in the difference image), and a large difference amount in non-flat regions with many edges (see the white parts in the difference image).

[0065] Then, the NeRF learning unit 103 performs NeRF learning using the difference image calculated in step S101 and the camera position and orientation input from the camera position and orientation input unit 101 as input (step S102).

[0066] In this case, NeRF learning of the differential image may be performed using a neural network completely separate from the NeRF model 104a. Alternatively, as shown in Fig. 10, new parameters (see "5) Map_Color" and "6) Map_VD" in the figure) may be added to the original NeRF model 104a to create a neural network that uses the same existence probability α as the NeRF model 104a, and transfer learning may be performed. When transfer learning is performed, the existence probability α of an object in three-dimensional space has already been learned, allowing learning to converge quickly.

[0067] In the example of FIG. ０ is the map basis information. １ is angle-dependent map basis information, which corresponds to the "VD component." １ is angle-dependent color information, and corresponds to a "basis function." For convenience of explanation, in FIG. 10, k is used as the VD component. １ Although only one component is listed here, two or more components may be used. In other words, the newly added parameter in the difference map model 110a indicates how the difference appears when the angle-independent map base information is viewed from various angles. This new parameter can be said to represent the difference before and after super-resolution processing as three-dimensional semi-transparent edge information using MPI.

[0068] Then, the NeRF learning unit 103 outputs the difference map model 110a, which is an NeRF model of the learned difference image, to the difference map model storage unit 110 and stores it (step S103).

[0069] At this time, the NeRF learning unit 103 converts the difference map model 110a into a format that can be rendered in real time and outputs it. For example, in the case of NeX, the NeRF learning unit 103 converts the color component k ０ and the color component k that changes depending on the viewing direction １ , k ２ , ..., k Ｎ and the corresponding weight basis function H １ , H ２ , ..., H Ｎ is output to the difference map model storage unit 110 in the MPI format.

[0070] Here, if reinforcement learning is performed by sharing the aforementioned existence probability α with the original NeRF model 104a, there is no need to newly output α. Furthermore, the resolution of the MPI in the difference map model 110a does not necessarily have to be the same as that of the original NeRF model 104a. For example, the resolution may be 1 / 2, 1 / 4, or less than the resolution of the original MPI.

[0071] <2-3. Processing Procedure of Real-Time Rendering Processing Using Switching Map> Next, the processing procedure of the real-time rendering processing using the switching map executed by the information processing device 100 will be described with reference to FIG. 8 already shown, and also with reference to FIGS. 11 to 13. FIG. 11 is a flowchart showing the processing procedure of the real-time rendering processing using the switching map executed by the information processing device 100. FIG. 12 is a diagram showing an example of a difference map. FIG. 13 is a diagram showing an example of a switching map.

[0072] 11 , in the real-time rendering process using the switching map, the NeRF estimation unit 105 estimates an NeRF output image and a difference map based on the NeRF model 104a in the NeRF model storage unit 104, the line of sight direction obtained from the line of sight direction input unit 113, and the difference map model 110a in the difference map model storage unit 110 (step S201). Then, the NeRF estimation unit 105 outputs the estimated NeRF output image to the NeRF output image storage unit 106 and the difference map to the difference map storage unit 111. An example of the estimated difference map is shown in FIG.

[0073] Next, the switching map generation unit 112 generates a super-resolution processing switching map from the difference map (step S202). First, the switching map generation unit 112 generates a switching map by reducing the resolution of the difference map for each image patch described above. An example of the switching map at this stage is shown in FIG. 13.

[0074] The switching map generation unit 112 then performs threshold processing on the switching map for each image patch in accordance with a threshold (e.g., the aforementioned 50%) previously set by the switching threshold setting unit 114, and sets the super-resolution processing to ON / OFF for each image patch. This corresponds to, for example, the "switching map (50% ON)" already shown in FIG. 5 . Note that the setting value of the map does not have to be a binary value of ON / OFF, but may be a setting value of three or more values ​​corresponding to, for example, first super-resolution processing / second super-resolution processing / OFF. Furthermore, the setting value of the map may indicate at least one of the processing target region and the non-processing target region.

[0075] Then, the super-resolution processing unit 107 performs super-resolution processing only on the valid areas (i.e., image patches set to ON) in the switching map in accordance with the setting values ​​of the switching map (step S203).

[0076] Finally, the super-resolution processing unit 107 combines the NeRF output image stored in the NeRF output image storage unit 106 with the super-resolution processed image (step S204). At this time, the super-resolution processing unit 107 performs the combination in accordance with the difference map stored in the difference map storage unit 111, and outputs the combined image to the super-resolution processed image storage unit 108.

[0077] The synthesis here may be performed by, for example, alpha blending as shown in the following equation (c) in accordance with the value of the difference map.

[0078] Here, x and y indicate the x and y coordinates of the image. out is the output image. δ is the difference map, nerf is the NeRF output image, and sr is the image after super-resolution processing. s() is a function that performs scaling to match the scales of nerf and sr, and f() is a function that associates the difference map with the alpha blend value (0 to 1). In areas where super-resolution processing has not been performed, out = s(nerf) is output. The actual image after synthesis corresponds to, for example, the "image after super-resolution processing (50%)" in Figure 5 already shown.

[0079] <<3. Modifications>> Of the processes described in the above-described embodiments of the present disclosure, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the process procedures, specific names, and information including various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0080] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0081] For example, the NeRF estimation unit 105 and the switching map generation unit 112 may be integrated into one processing unit.

[0082] The above-described embodiments of the present disclosure can be combined as appropriate within the scope of the present disclosure without causing any contradiction in the processing content. The order of the steps shown in the sequence diagrams or flowcharts of the present embodiments can be changed as appropriate.

[0083] In the above-described embodiment of the present disclosure, the super-resolution processed image is displayed on a display device such as a monitor. However, the display may be, for example, a light field display that provides stereoscopic vision using parallax images. In this case, the gaze direction input unit 113 inputs the user's gaze direction from, for example, a camera mounted on the light field display. Furthermore, the NeRF model 104a and the difference map model 110a render a left-eye image for the user's left eye and a right-eye image for the user's right eye, respectively.

[0084] <<4. Hardware Configuration>> The information processing device 100 according to the embodiment of the present disclosure described above is realized by, for example, a computer 1000 configured as shown in Fig. 14. Fig. 14 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing device 100. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, a secondary storage device 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0085] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the secondary storage device 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the secondary storage device 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0086] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0087] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records the program according to this embodiment, which is an example of the program data 1450.

[0088] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550. For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0089] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0090] For example, when the computer 1000 functions as the information processing device 100, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to realize the functions of the control unit described above. The secondary storage device 1400 stores the program according to this embodiment, various models including the NeRF model 104a and the difference map model 110a, and various data including the NeRF output image and the image after super-resolution processing. The CPU 1100 reads and executes the program data 1450 from the secondary storage device 1400. Alternatively, the CPU 1100 may acquire these programs from another device via an external network 1550.

[0091] <<5. Conclusion>> As described above, according to one embodiment of the present disclosure, the information processing device 100 includes the NeRF estimation unit 105 (corresponding to an example of an “estimation unit”) and the super-resolution processing unit 107. The NeRF estimation unit 105 estimates an NeRF output image (corresponding to an example of a “new viewpoint image”) according to an arbitrary line of sight based on a multi-viewpoint image including a plurality of two-dimensional images using the NeRF model 104a (corresponding to an example of a “first model”) that has been trained to estimate the NeRF output image according to an arbitrary line of sight. The super-resolution processing unit 107 performs super-resolution processing on the estimated NeRF output image. Furthermore, the NeRF estimation unit 105 estimates a processing target region to be subjected to the super-resolution processing and a non-processing target region to be not subjected to the super-resolution processing for each NeRF output image using the difference map model 110a (corresponding to an example of a “second model”) that has been trained to estimate the processing target region and the non-processing target region to be subjected to the super-resolution processing according to the line of sight. Furthermore, the super-resolution processing unit 107 performs the super-resolution processing on the estimated processing target region, but does not perform the super-resolution processing on the estimated non-processing target region, thereby enabling the speed of the super-resolution processing to be increased when the super-resolution processing is performed on the rendering result of the new viewpoint image.

[0092] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0093] Furthermore, the effects of each embodiment described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0094] Note that the present technology may also be configured as follows: (1) An information processing device comprising: an estimation unit that estimates a new viewpoint image using a first model that has been trained to estimate the new viewpoint image according to an arbitrary line of sight direction based on a multi-viewpoint image including a plurality of two-dimensional images; and a super-resolution processing unit that performs super-resolution processing on the estimated new viewpoint image, wherein the estimation unit estimates a processing target region to be subjected to the super-resolution processing and a non-processing target region to be not subjected to the super-resolution processing for each of the new viewpoint images using a second model that has been trained to estimate, according to the line of sight direction, the processing target region and the non-processing target region, and the super-resolution processing unit performs the super-resolution processing on the estimated processing target region and does not perform the super-resolution processing on the estimated non-processing target region. (2) The information processing device according to (1), further comprising: a learning unit that learns the first model based on the multi-viewpoint images, and learns the second model based on a difference image between the new viewpoint image and an image after the super-resolution processing for each viewpoint of the plurality of two-dimensional images. (3) The information processing device according to (2), wherein the estimation unit estimates the difference image according to the line of sight using the second model and generates map information indicating at least one of the processing target region or the non-processing target region based on the estimated difference image, and the super-resolution processing unit performs the super-resolution processing on the processing target region indicated by the map information. (4) The information processing device according to (3), wherein the estimation unit generates the map information in which a region in the difference image estimated using the second model where the difference amount is relatively large is set as the processing target region and a region in the difference image estimated using the second model where the difference amount is relatively small is set as the non-processing target region. (5) The information processing device according to (4), wherein the estimation unit sets the processing target region and the non-processing target region for each predetermined pixel unit. (6) The information processing device according to any one of (2) to (5), wherein the estimation unit sets the processing target region based on a threshold indicating a proportion of the difference image estimated using the second model to be set as the processing target region.(7) The information processing device according to any one of (2) to (6), wherein the learning unit learns the second model by transfer learning based on the first model. (8) The information processing device according to (7), wherein the first model and the second model cache intermediate data in the form of a multi-layer image in which a plurality of images are arranged in a depth direction from a viewpoint position of the two-dimensional image, and share a probability of an object existing at an intersection between a ray tracked from the line of sight and the multi-layer image. (9) The information processing device according to any one of (1) to (8), wherein the first model and the second model are models for synthesizing a radiance field using a neural network. (10) The information processing device according to any one of (1) to (9), wherein the super-resolution processing unit synthesizes the image after the super-resolution processing and the new viewpoint image and outputs the synthesized image. (11) An information processing method comprising: estimating a new viewpoint image using a first model trained to estimate the new viewpoint image according to an arbitrary line of sight based on a multi-viewpoint image including a plurality of two-dimensional images; performing super-resolution processing on the estimated new viewpoint image; estimating a processing target region to be subjected to the super-resolution processing and a non-processing target region to be not subjected to the super-resolution processing for each new viewpoint image using a second model trained to estimate the processing target region and the non-processing target region to be not subjected to the super-resolution processing according to the line of sight; performing the super-resolution processing on the estimated processing target region; and not performing the super-resolution processing on the estimated non-processing target region.(12) A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following: estimating a new viewpoint image using a first model that has been trained to estimate the new viewpoint image according to an arbitrary line of sight based on a multi-viewpoint image including a plurality of two-dimensional images; performing super-resolution processing on the estimated new viewpoint image; estimating a processing target region to be subjected to the super-resolution processing and a non-processing target region to be not subjected to the super-resolution processing for each new viewpoint image using a second model that has been trained to estimate the processing target region and the non-processing target region to be not subjected to the super-resolution processing according to the line of sight; performing the super-resolution processing on the estimated processing target region; and not performing the super-resolution processing on the estimated non-processing target region.

[0095] 50 Camera 100 Information processing device 101 Camera position and orientation input unit 102 Multi-viewpoint image input unit 103 NeRF learning unit 104 NeRF model storage unit 104a NeRF model 105 NeRF estimation unit 106 NeRF output image storage unit 107 Super-resolution processing unit 108 Super-resolution processed image storage unit 109 Difference image calculation unit 110 Difference map model storage unit 110a Difference map model 111 Difference map storage unit 112 Switching map generation unit 113 Gaze direction input unit 114 Switching threshold setting unit α Existence probability

Claims

1. An information processing device comprising: an estimation unit that estimates a new viewpoint image using a first model that has been trained to estimate a new viewpoint image according to an arbitrary line of sight based on a multi-view image including a plurality of two-dimensional images; and a super-resolution processing unit that performs super-resolution processing on the estimated new viewpoint image, wherein the estimation unit estimates the processing target area and the non-processing target area using a second model that has been trained to estimate, for each new viewpoint image, a processing target area where the super-resolution processing is performed and a non-processing target area where the super-resolution processing is not performed according to the line of sight, and the super-resolution processing unit performs the super-resolution processing on the estimated processing target area and does not perform the super-resolution processing on the estimated non-processing target area.

2. The information processing device according to claim 1, further comprising a learning unit that learns the first model based on the multi-viewpoint image and learns the second model based on a difference image between the new viewpoint image for each viewpoint of the multiple two-dimensional images and the image after the super-resolution processing.

3. The information processing device described in claim 2, wherein the estimation unit estimates the difference image according to the gaze direction using the second model, and generates map information indicating at least one of the processing target area or the non-processing target area based on the estimated difference image, and the super-resolution processing unit performs the super-resolution processing on the processing target area indicated by the map information.

4. The information processing device according to claim 3, wherein the estimation unit generates the map information by setting an area in the difference image estimated using the second model where the difference amount is relatively large as the processing target area, and setting an area in the difference image estimated using the second model where the difference amount is relatively small as the non-processing target area.

5. The information processing device according to claim 4, wherein the estimation section sets the processing target area and the non-processing target area for each predetermined pixel unit.

6. The information processing device according to claim 2, wherein the estimation unit sets the processing target area based on a threshold value indicating a proportion of the processing target area in the difference image estimated using the second model.

7. The information processing device according to claim 2, wherein the learning unit learns the second model by transfer learning based on the first model.

8. The information processing device according to claim 7, wherein the first model and the second model cache intermediate data in the form of a multi-layered image arranged in a depth direction from the viewpoint position of the two-dimensional image, and share the probability of the existence of an object at the intersection between a ray tracked from the line of sight and the multi-layered image.

9. The information processing device according to claim 1, wherein the first model and the second model are models for synthesizing a radiance field using a neural network.

10. The information processing device according to claim 1, wherein the super-resolution processing unit synthesizes the image after the super-resolution processing and the new viewpoint image and outputs the synthesized image.

11. An information processing method comprising: estimating a new viewpoint image using a first model trained to estimate a new viewpoint image according to an arbitrary line of sight based on a multi-view image including a plurality of two-dimensional images; performing super-resolution processing on the estimated new viewpoint image; estimating a processing target area where the super-resolution processing is performed and a non-processing target area where the super-resolution processing is not performed for each new viewpoint image according to the line of sight, estimating the processing target area and the non-processing target area using a second model trained to estimate the super-resolution processing on the estimated processing target area and the non-processing target area where the super-resolution processing is not performed on the estimated non-processing target area; performing the super-resolution processing on the estimated processing target area, and not performing the super-resolution processing on the estimated non-processing target area.

12. A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following operations: estimating a new viewpoint image using a first model that has been trained to estimate a new viewpoint image according to an arbitrary line of sight based on a multi-view image including a plurality of two-dimensional images; performing super-resolution processing on the estimated new viewpoint image; estimating a processing target area where the super-resolution processing is performed and a non-processing target area where the super-resolution processing is not performed for each new viewpoint image according to the line of sight, estimating the processing target area and the non-processing target area where the super-resolution processing is not performed, performing the super-resolution processing on the estimated processing target area, and not performing the super-resolution processing on the estimated non-processing target area.

Images