Information processing device, information processing method, and computer-readable non-transitory storage medium

Abstract

This information processing device has a foreground / background separation image generation unit and a super-resolution processing unit. The foreground / background separation image generation unit acquires, as a foreground image and a background image, images of a foreground part and a background part that overlap each other in the sightline direction. The super-resolution processing unit acquires a motion vector for motion compensation of the foreground image and a motion vector for motion compensation of the background image on the basis of motion in the foreground part and motion in the background part, respectively.

Description

Information processing device, information processing method, and computer-readable non-temporary storage medium

[0001] The present invention relates to an information processing apparatus, an information processing method, and a computer-readable non-temporary storage medium.

[0002] MFSR (Multi-Frame Super Resolution) is a well-known DNN (Deep Neural Network) processing technique for video. MFSR is a method that blends information from the current frame with recursive information from past frames and uses the blended result to perform super-resolution on the current frame.

[0003] Special Publication No. 2022-547517

[0004] Recursive information from past frames is used as input to the DNN along with the current frame. To maintain consistency with the current frame, motion compensation is performed on the recursive information using motion vectors. However, when the background is seen through the foreground, such as when the foreground is transparent, two movements occur at the see-through position, corresponding to the foreground and the background. The motion vector is calculated, for example, as an average of the two movements, or only one of the movements is used. As a result, proper motion compensation may not be possible, and multiple images may occur at the see-through position.

[0005] Therefore, this disclosure proposes an information processing device, an information processing method, and a computer-readable non-temporary storage medium capable of suppressing multiple images.

[0006] According to this disclosure, an information processing device is provided, comprising: a foreground-background separation image generation unit that acquires images of a foreground and background that overlap in the line of sight as a foreground image and a background image, respectively; and a super-resolution processing unit that acquires motion vectors for motion compensation of the foreground image and motion vectors for motion compensation of the background image, respectively, based on the movement in the foreground and the movement in the background independent of the foreground. Furthermore, according to this disclosure, an information processing method is provided in which the information processing of the information processing device is performed by a computer, and a computer-readable non-temporary storage medium is provided that stores a program that enables the computer to implement the information processing of the information processing device.

[0007] This figure shows an example of a method for generating new viewpoint images. This is an image diagram of MPI. This is a conceptual diagram of equation (b). This is a supplementary explanatory diagram of Figure 3. This figure shows yet another example of a method for generating new viewpoint images. This is an explanatory diagram of MFSR. This figure shows an example of MPI that is the target of super-resolution processing. This figure shows MPI imaged by depth and separated into foreground and background parts. This figure shows a comparison of output images with and without foreground and background separation. This is a block diagram showing an example of the configuration of an information processing device. This figure shows the processing procedure of super-resolution processing with foreground and background separation. This figure shows an example of generating an image focused on a balloon while changing the angle of the virtual camera. This figure shows an example of the hardware configuration of an information processing device.

[0008] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0009] The explanation will proceed in the following order: [1. Method for generating new viewpoint images] [1-1. Generating new viewpoint images using NeRF] [1-2. Generating new viewpoint images using NeX] [1-3. NeRF with video capabilities] [1-4. Generating new viewpoint images using 3DGS] [1-5. 3DGS with video capabilities] [2. Challenges in super-resolution processing and real-time rendering] [3. Accelerating super-resolution processing using MFSR] [4. Example configuration of information processing device] [4-1. Block diagram] [4-2. Processing procedure] [5. Example hardware configuration] [6. Effects]

[0010] [1. Method for generating new viewpoint images] [1-1. Generation of new viewpoint images using NeRF] Figure 1 shows an example of a method for generating new viewpoint images.

[0011] Figure 1 shows an example of generating a new viewpoint image using NeRF (Neural Radiance Fields). NeRF is a technology that takes a set of images of a target scene taken from multiple viewpoints (multi-viewpoint images) as input and acquires a 3D representation through AI-based processing (see Non-Patent Literature 1 below). The target scene refers to the scene that NeRF will infer. In the example in Figure 1, a scene of a cat sitting on the floor is depicted as the target scene.

[0012] [Non-Patent Document 1] Mildenhall, Ben, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.” arXiv [cs.CV]. arXiv. http: / / arxiv.org / abs / 2003.08934.

[0013] NeRF performs deep learning (NeRF learning) using sparse viewpoint images captured by camera 50 to optimize the parameters of the radiance field. NeRF reconstructs the radiance field using an inference model called "continuous volumetric scene function" (NeRF model 104a) and infers a new viewpoint image by reproducing the target scene from an arbitrary viewpoint (NeRF estimation). The reconstruction of the radiance field is performed using a technique called volume rendering.

[0014] For example, when viewpoint and line-of-sight information is input to the NeRF model 104a, an image of the target scene observed along the line-of-sight direction from that viewpoint is inferred. The NeRF model 104a is implemented as a neural network that takes viewpoint and line-of-sight direction as input information and outputs a new viewpoint image seen from that viewpoint and line-of-sight direction. The arrows in the image obtained by NeRF estimation (NeRF output image) schematically represent how the position and orientation of the object (in this case, the cat) change in accordance with changes in the viewpoint position and line-of-sight direction.

[0015] It is known that images generated by NeRF are blurrier than images from camera 50 used for training. Therefore, super-resolution processing is applied to NeRF output images. Super-resolution processing refers to the process of increasing the resolution of an image using generative AI such as GAN (Generative Adversarial Networks). By combining NeRF and super-resolution processing, high-resolution rendering images can be obtained in a relatively short time.

[0016] [1-2. Generating Novel Viewpoint Images Using NeX] Figures 2 to 4 illustrate other examples of generating novel viewpoint images.

[0017] This example demonstrates the generation of new viewpoint images using NeX. While NeRF can render high-resolution new viewpoint images, volume rendering using neural networks requires a significant amount of processing time. Specifically, even when processing one viewpoint using a GPU (Graphical Processing Unit), it takes approximately 30 seconds.

[0018] In response to this, many techniques have recently been proposed to shorten rendering time by caching the computation results of neural networks in various formats. In a method called "NeX" disclosed in Non-Patent Document 2 below, intermediate data of the NeRF model 104a is cached in the form of multi-plane images (MPI) arranged in multiple layers in the depth direction. As a result, volume rendering processing is sped up by approximately 1000 times compared to NeRF, enabling real-time rendering.

[0019] [Non-patent Document 2] Wizadwongsa, Suttisak, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn. 2021. “NeX: Real-Time View Synthesis with Neural Basis Expansion.” arXiv [cs.CV]. arXiv. http: / / arxiv.org / abs / 2103.05606.

[0020] Figure 2 is an illustrative diagram of the MPI. The MPI has a structure consisting of a group of RGBA layers in which semi-transparent RGBA images are stacked in a frustum-like shape in three-dimensional space to cover a depth from the vicinity of the camera 50 to a sufficiently distant point. A new viewpoint image is generated by overlapping each RGBA image constituting the MPI while shifting it according to the line of sight direction from each viewpoint. In Figure 2, the "first viewpoint" is, for example, the shooting position of the camera 50. The "second viewpoint" is, for example, a new arbitrary viewpoint position other than the shooting position of the camera 50.

[0021] In NeX, the radiance field is learned to optimize the RGB values ​​obtained by volume rendering of the MPI so that they approach the RGB values ​​captured by camera 50. In NeX's rendering process, a light ray is tracked from the line of sight, and the probability of existence "α" at the point where the light ray intersects with the MPI is multiplied by the color of that point and added. Equation (a) below is given by the probability of existence A (= {α １ , α ２ , ..., αＤ}) and the color component C(={c １ , c ２ , …, c Ｄ}) as input, which represents the calculation method of the RGB values of the output image.

[0022]

[0023] In the formula (a), "c ｄ " is calculated by weighted addition of the angle-independent color component "k ０ ", which does not change with the viewing angle, and the view-dependent (hereinafter, appropriately referred to as "VD") color components "k １ " to "k N " that change with the viewing angle. The following formula (b) is the formula for calculating "c ｄ ".

[0024]

[0025] FIG. 3 is a conceptual diagram of the formula (b). FIG. 4 is a supplementary explanatory diagram of FIG. 3. As shown in FIG. 3, any one point (here, the upper right corner point) of a certain layer in the NeRF model 104a of NeX has the reflection coefficients consisting of the above "k ０ " and the "k ０ " at various angles for each existence probability "α" at such a point. １ ", …, "k Ｎ ".

[0026] The basis functions "H １ ", …, "H Ｎ " learned from the neural network are weighted and linearly combined by "k １ ", …, "k Ｎ " according to the viewing angle. Thereby, the RGB values at the above-mentioned one point when viewed at a certain angle are determined. By such modeling, the phenomenon that the appearance of an object is different depending on the angle even at the same location can be reproduced.

[0027] Although there is some overlap in the explanation, to supplement the explanation of each parameter shown in Figure 3 using Figure 4, the aforementioned "α" is the probability of existence in three-dimensional coordinates and corresponds to the "geometric component" (see "1) Alpha" in the figure). Also, "k ０ This is angle-independent color information and corresponds to the "texture component" (see "2) Color" in the diagram).

[0028] Also, "k １ "~"k ４ "...is angle-dependent color information and corresponds to the "VD component" (see "3) Color_VD" in the figure). Also, "H １ "~"H ４ "...represents angle-dependent color information and corresponds to a "basis function" (see "4) Basis" in the diagram).

[0029] [1-3. Video-Compatible NeRF] Conventional NeRF is suitable for reproducing stationary objects and scenes, and does not support dynamic scenes. However, in recent years, video-compatible NeRF (Dynamic NeRF), which has evolved to support dynamic scenes and videos, has been proposed. This makes it possible to reproduce scenes that change over time and moving objects. The following technologies have been introduced to reproduce dynamic scenes with high accuracy and efficiency.

[0030] - Introduction of spatiotemporal representation: A spatiotemporal model that takes the time dimension into account is used to reproduce dynamic scenes. - Maintaining continuity between frames: Constraints are imposed to maintain continuity between frames of the video. - Tracking and reproducing dynamic objects: Dynamic objects are represented by tracking moving objects and changing environments and deforming stationary objects.

[0031] Video-compatible NeRF enables real-time display of scenes, such as moving people, animals, and changing landscapes, as the viewing perspective changes. As a result, it is expected to be applied in fields such as VR (Virtual Reality), AR (Augmented Reality), and movie production. A recent survey paper on video NeRF (see Non-Patent Document 3 below) introduced a list of representative papers from 2021 to 2023 when video-compatible NeRF began to emerge.

[0032] [Non-Patent Document 3] Dynamic NeRF: A Review, https: / / arxiv.org / pdf / 2405.08609

[0033] [1-4.3DGS for Generating New Viewpoint Images] Figure 5 is a diagram showing yet another example of a method for generating a new viewpoint image.

[0034] This example is an example of generating a new viewpoint image using 3D Gaussian Splating (3DGS). 3DGS is a new technology for representing 3D scenes and reconstructs 3D scenes using deep learning, similar to NeRF. GS represents the scene using the Gaussian distribution GA. By applying a Gaussian to each sampling point CO, GS generates a smooth and highly accurate 3D model (see Non-Patent Document 4 below).

[0035] [Non-Patent Document 4] PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics, https: / / arxiv.org / pdf / 2311.12198

[0036] 3DGS is a method similar to NeRF but differs from NeRF in the following points. First, they differ in the method of data representation. For example, NeRF represents the scene as a volumetric radiance field and learns the color and density of each sampling point CO through a neural network. On the other hand, 3DGS represents the scene as a collection of a large number of Gaussian distributions GA, and each Gaussian distribution GA has its position, shape, and color.

[0037] The rendering mechanisms of the two also differ. For example, NeRF calculates color and density through a neural network for each viewpoint and generates an image based on that. On the other hand, 3DGS directly renders using information from a Gaussian distribution GA that makes up the scene, and generates an image by projecting a Gaussian distribution GA according to the viewpoint.

[0038] The two methods also differ in computational efficiency. For example, NeRF often takes a long time to render. Real-time rendering methods (such as NeX) often use techniques that cache radiance field information in an explicit representation. On the other hand, 3DGS utilizes the characteristics of a Gaussian distribution (GA), resulting in high computational efficiency and suitability for real-time rendering. Intuitively, NeRF data is represented as points of uniform density, while GS data is represented as spheres with width. Therefore, GS requires fewer data points to create an image, resulting in faster rendering.

[0039] [1-5. Video-Compatible 3DGS] To reproduce scenes and dynamic objects that change over time, video-compatible 3DGS (Dynamic 3D Gaussian Platting), an extension of 3DGS technology, has been proposed since 2024. The following technologies have been introduced to reproduce dynamic scenes with high accuracy and efficiency.

[0040] • Spatiotemporal Gaussian Model: A spatiotemporal Gaussian model with an added time dimension is used to represent dynamic scenes. • Maintaining Continuity Between Frames: Tracking and interpolation techniques of a time-direction Gaussian distribution GA are used to maintain continuity between frames. • Real-Time Processing: Real-time rendering of dynamic scenes is achieved through highly efficient computation algorithms and hardware optimizations.

[0041] Video-enabled 3DGS can display scenes including moving people or animals, or changing landscapes, in real time as the viewpoint changes. One representative paper is Non-Patent Document 5, which shows that video-enabled 3DGS is faster and produces higher image quality than conventional video-enabled NeRF and similar existing methods.

[0042] [Non-patent Document 5] 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering, https: / / arxiv.org / abs / 2310.08528

[0043] [2. Challenges in Super-Resolution Processing and Real-Time Rendering] NeRF and 3DGS have the challenge of producing output images that are blurrier or contain artifacts compared to the input image, making it difficult to reproduce sharp images like those in real life. As a signal processing solution, applying learning-based super-resolution technology to the output image of NeRF makes it possible to restore sharper images that are closer to the input image.

[0044] For example, a method has been proposed that combines Direct Voxel Grid Optimization, a NeRF technique, with learning-based super-resolution processing, and it has been shown that this method can produce sharper output images than conventional NeRF (see Non-Patent Literature 6 below). However, super-resolution processing takes time, which presents a challenge in that it makes real-time rendering (e.g., 60 fps or more) at high resolutions (e.g., 4K or 8K) difficult.

[0045] [Non-Patent Document 6] Wang, Zhongshu, Lingzhi Li, Zhen Shen, Li Shen, and Liefeng Bo. 2022. “4K-NeRF: High Fidelity Neural Radiance Fields at Ultra High Resolutions.” arXiv [cs.CV]. arXiv. http: / / arxiv.org / abs / 2212.04701

[0046] [3. Accelerating Super-Resolution Processing with MFSR] As a method to accelerate super-resolution processing, a method has been proposed that uses the super-resolution result of the previous frame to perform super-resolution. This method is called MFSR (Multi-frame super-resolution). Figure 6 is an explanatory diagram of MFSR.

[0047] MFSR is a technology for improving the quality of video and film. MFSR generates high-resolution images using multiple frames. The high-resolution inferred image obtained by applying super-resolution processing to past frames is used as input to a DNN along with the current frame. To ensure consistency with the current frame, motion compensation based on motion vectors is applied to the inferred image of past frames.

[0048] A motion vector is a vector that indicates the movement of an object between frames (the amount and direction of pixel movement). Motion compensation is the process of predicting post-motion data from pre-motion data. For example, if motion compensation is applied to an inferred image of a past frame, the image predicted at the current time is obtained as the motion-compensated inferred image. The DNN receives the motion-compensated inferred image along with the current frame as input.

[0049] Super-resolution processing that does not use past information (single-frame super-resolution) generates a high-resolution image from a single image (the current frame). On the other hand, MFSR utilizes multiple frames that are consecutive in time. Therefore, MFSR can extract more information than single-frame super-resolution and produce a higher-definition image. An example of MFSR is DLSS (Deep Learning Super Sampling) provided by Nvidia.

[0050] Motion vectors are set for each pixel. Each pixel is assigned one motion vector. However, when the foreground is transparent, and the background is visible through the foreground, two movements occur at the visible point, corresponding to the foreground and the background. In this case, the motion vector is calculated as an average of the two movements. As a result, a movement different from both the foreground and the background is calculated, making it impossible to perform proper motion compensation. Consequently, problems such as multiple images may occur at the visible point.

[0051] For example, in rendering an object with a faint shadow, the movement of the shadow itself and the movement of the object it's projected onto overlap at the projection location. In rendering semi-transparent particles, the movement of objects behind the particles can be seen through them. In a drawing of moving fine hairs, the movement of objects behind them can be seen through the gaps in the hair. Even in scenes where objects in the foreground are moving quickly, it can sometimes appear as if objects in the background are visible through the hair.

[0052] In this disclosure, the state in which the background is visible through the foreground is referred to as the foreground being "transparent." The "foreground" refers to objects closer to the viewpoint VP of the virtual camera performing the rendering. The "background" refers to objects further away from the viewpoint VP of the virtual camera. "Objects" also include optical images that do not have a physical body, such as shadows.

[0053] When the foreground is semi-transparent, it allows at least a portion of the background to be seen through to the foreground along the line of sight. The "semi-transparent" state is mainly due to the optical or structural properties of the foreground. However, in this disclosure, a state in which the background appears to be pseudo-transparent due to the rapid movement of the foreground is also referred to as "semi-transparent." In other words, the foreground can allow at least a portion of the background to be seen through in the line of sight due to its transparency or light-transmitting structure, or due to the movement of the foreground.

[0054] Examples of "translucent" objects include (A) objects made of materials that transmit visible light (such as glass or transparent balloons), (B) objects whose background is visible through light-transmitting structures such as tiny grooves, holes, or gaps between aggregates (such as grids, meshes, or hair), and (C) objects whose background is revealed or hidden by rapid movement.

[0055] When a translucent foreground moves differently from the background, two separate motion vectors are set at the overlapping position of the foreground and background, each representing a different movement. Since the motion vector is calculated as an average of the two movements, appropriate motion compensation is not performed at the overlapping position. Therefore, the following method is employed in this disclosure.

[0056] First, separate motion compensation and super-resolution processing are performed on the foreground and background according to their respective motion vectors. The independently super-resolution processed foreground and background images are combined and output as the final inference image (super-resolution processed image). In this method, appropriate motion compensation is performed on both the foreground and background. Therefore, multiple images are less likely to occur at the overlapping location of the foreground and background.

[0057] Motion compensation and super-resolution processing can be performed on intermediate data cached in MPI format. Figure 7 shows an example of MPI to be subjected to super-resolution processing. The left side of Figure 7 shows intermediate data for the NeRF model. The right side of Figure 7 shows intermediate data for 3DGS. Figure 8 shows MPI images separated by depth, with the foreground (FG) and background (BG) separated.

[0058] Let's consider the case where NeRF is used to infer an image viewed from the viewpoint VP. In this case, from the multiple layer PIs (RGBA images) that make up the MPI, one or more layer PIs representing the foreground FG and one or more layer PIs representing the background BG are separated and extracted. The foreground image is inferred by reconstructing the one or more layer PIs representing the foreground FG. The background image is inferred by reconstructing the one or more layer PIs representing the background BG (foreground-background separation).

[0059] If the movement of the background (BG) as seen through the foreground (FG) differs significantly from the movement of the foreground (FG), motion compensation and super-resolution processing are performed independently on the foreground and background images. The difference in movement between the foreground (FG) and background (BG) can be determined based on the motion vectors of the foreground (FG) and background (BG).

[0060] The same applies when inferring new viewpoint images using 3DGS. In NeRF, the radiance field is represented by the color and density of each sampling point CO, while in 3DGS, it is represented by the Gaussian distribution GA of each sampling point CO. Aside from these points, the calculation methods of NeRF can also be applied to 3DGS. Both NeRF and 3DGS are capable of handling video. For example, in NeRF, a video 3D model can be represented by changing the layer PI over time. In 3DGS, a video 3D model can be represented by changing the Gaussian distribution GA over time.

[0061] Furthermore, factors that may cause a difference in the movement of the foreground (FG) and background (BG) include (i) a large difference in depth between the foreground (FG) and background (BG), and (ii) the 3D models of the foreground (FG) and background (BG) exhibiting different movements. As mentioned in (i) above, differences in movement caused by the difference in depth between the foreground (FG) and background (BG) can occur even with still image 3D models. On the other hand, even if the difference in depth between the foreground (FG) and background (BG) is small, in the case of moving 3D models where the 3D model itself is moving, a difference in the movement of the foreground (FG) and background (BG) will occur, and appropriate motion compensation will not be performed.

[0062] Figure 9 shows a comparison of output images with and without foreground-background separation. The left side of Figure 9 shows an example with foreground-background separation. The right side of Figure 9 shows an example without foreground-background separation.

[0063] Figure 9 shows a scene where a chair in front of a desk is moved quickly. The rapid movement of the chair creates a "semi-transparent" effect, making the desk and wall behind it appear translucent. In the example without foreground-background separation (right side of Figure 9), the chair appears significantly blurred. In the example with foreground-background separation (left side of Figure 9), the blur is reduced. This demonstrates that foreground-background separation improves blur caused by multiple images.

[0064] In the above explanation, we assumed there were two objects to be separated. However, the number of objects to be separated is not limited to two. Even if there are three or more objects to be separated, the image can be separated using the same method as described above, and motion compensation and super-resolution processing can be performed individually on each image.

[0065] [4. Example of Information Processing Device Configuration] [4-1. Block Diagram] Super-resolution processing with foreground-background separation can be performed using an information processing device 1 as shown in Figure 10. Figure 10 is a block diagram showing an example of the configuration of the information processing device 1. An example of information processing using NeRF will be described below, but the method of this disclosure is similarly applicable to other methods such as 3DGS.

[0066] The information processing device 1 renders moving subjects using the video-compatible NeRF method. For example, the information processing device 1 includes a camera position and orientation input unit 11, a multi-view image input unit 12, a learning unit 13, an inference model storage unit 14, a new viewpoint inference unit 15, an inference image storage unit 16, a super-resolution processing unit 17, a super-resolution processed image storage unit 18, a foreground-background separated image generation unit 19, a foreground-background separated image storage unit 20, a gaze direction input history storage unit 21, and a gaze direction input unit 22.

[0067] The multi-view image input unit 12 acquires a set of images (multi-view images) of the target scene taken from various viewpoints. The camera position and orientation input unit 11 acquires pose information of the camera 50 that captured the target scene. The pose information includes information about the position and orientation of the camera 50. The learning unit 13 performs deep learning (NeRF learning) using the multi-view images and pose information to optimize the parameters of the radiance field. The inference model storage unit 14 saves the radiance field model obtained through learning as an inference model (NeRF model 104a).

[0068] The gaze direction input unit 22 acquires pose information from the virtual camera. The gaze direction input history storage unit 21 stores the input history of pose information. When pose information is input, the inference model infers an image of the target scene as seen from the viewpoint and gaze direction indicated by the pose information. The new viewpoint inference unit 15 reconstructs the radiance field using the inference model. As a result, the new viewpoint inference unit 15 infers an image of the target scene reproduced from an arbitrary viewpoint as the new viewpoint image (NeRF estimation).

[0069] The inference image storage unit 16 stores the new viewpoint image obtained from the inference model as an inference image. The super-resolution processing unit 17 performs super-resolution processing on the inference image using the MFSR method. The super-resolution processing unit 17 acquires the super-resolution processed inference image as a super-resolution processed image. The super-resolution processed image storage unit 18 stores the super-resolution processed image.

[0070] The foreground-background separation image generation unit 19 separates and generates images of the foreground FG and background BG (foreground-background separation). For example, the foreground-background separation image generation unit 19 separates and acquires images of the foreground FG and background BG when the movements of the foreground FG and background BG differ significantly beyond an acceptable standard. The foreground-background separation image generation unit 19 acquires images of the foreground FG and background BG that overlap in the line of sight as the foreground image and background image, respectively. The criteria for determining whether or not foreground-background separation is necessary (acceptable standard) can be arbitrarily set by the system developer.

[0071] The foreground-background separation image generation unit 19 can extract foreground and background images from intermediate data of the radiance field cached in Multiplane Image (MPI) format. For example, the foreground-background separation image generation unit 19 calculates the motion vector of each sampling point CO of the radiance field. Based on the discontinuous distribution of the motion vectors, the foreground-background separation image generation unit 19 identifies the foreground FG and background BG to be separated.

[0072] "Discontinuous motion vectors" means that the motion vectors change significantly beyond an acceptable threshold. The foreground-background separation image generation unit 19 separates and generates foreground and background images from the MPI when the movement of the foreground part FG indicated by the motion vector for the foreground image and the movement of the background part BG indicated by the motion vector for the background image differ significantly beyond an acceptable threshold. The threshold (acceptable threshold) for discontinuity in motion vectors can be arbitrarily set by the system developer.

[0073] For example, the foreground-background separation image generation unit 19 detects the amount of change in the gaze direction from the input history of pose information. The foreground-background separation image generation unit 19 acquires the movement of each sampling point CO based on the amount of change in the gaze direction. If the subject is moving, the foreground-background separation image generation unit 19 acquires the movement of each sampling point CO according to the movement of the subject. Based on the movement of each sampling point CO, the foreground-background separation image generation unit 19 detects the difference in movement between the foreground FG and the background BG.

[0074] The foreground-background separated image storage unit 20 stores the foreground image and background image obtained by foreground-background separation. The super-resolution processing unit 17 performs super-resolution processing on the separated foreground image and background image individually. For example, the super-resolution processing unit 17 obtains motion vectors for motion compensation of the foreground image and motion vectors for motion compensation of the background image, respectively, based on the motion in the foreground part FG and the motion in the background part BG which is independent of the foreground part FG. The super-resolution processing unit 17 performs super-resolution processing on the foreground image and background image using the MFSR method.

[0075] For example, the super-resolution processing unit 17 uses motion vectors for the foreground image and motion vectors for the background image to perform super-resolution processing on the foreground image and background image separately. The super-resolution processing unit 17 then combines the super-resolution processed foreground image and background image. The super-resolution processing unit 17 acquires the combined image as a super-resolution processed image viewed from the direction of the line of sight.

[0076] When performing super-resolution processing on a foreground image, the super-resolution processing unit 17 uses a motion vector for the foreground image to perform motion compensation on the foreground image of past frames. When performing super-resolution processing on a background image, the super-resolution processing unit 17 uses a motion vector for the background image to perform motion compensation on the background image of past frames.

[0077] The foreground-background separation image generation unit 19 does not perform foreground-background separation if the difference between the movement of the foreground part FG and the movement of the background part BG meets the acceptable criteria. As described above, the new viewpoint inference unit 15 superimposes multiple layers PI that constitute the MPI and infers an image viewed from the direction of the line of sight. The foreground-background separation image generation unit 19 outputs the inferred image estimated by the new viewpoint inference unit 15 directly to the inferred image storage unit 16.

[0078] [4-2. Processing Procedure] Figure 11 shows the processing procedure for super-resolution processing with foreground-background separation.

[0079] The foreground-background separation image generation unit 19 calculates the amount of movement of each sampling point CO that occurs during one frame period for each layer PI arranged in the depth direction (step S1). For example, the foreground-background separation image generation unit 19 calculates the movement of the sampling CO as a combination of the movement of the sampling CO caused by changes in viewpoint and line of sight direction and the movement of the sampling CO caused by the movement of the subject.

[0080] The foreground-background separation image generation unit 19, when there is a large difference in the amount of motion of the sampling points between the layer PIs, extracts one or more layer PIs representing the foreground FG and one or more layer PIs representing the background BG from the multiple layer PIs that make up the MPI. The foreground-background separation image generation unit 19 generates a foreground image by performing volume rendering on the one or more layer PIs representing the foreground FG. The foreground-background separation image generation unit 19 generates a background image by performing volume rendering on one or more layer PIs representing the background BG (step S2).

[0081] The super-resolution processing unit 17 performs super-resolution processing independently on the foreground image and background image using the MFSR method (step S3). For example, the super-resolution processing unit 17 acquires a motion vector indicating the movement of the foreground part FG as a motion vector for the foreground image. The super-resolution processing unit 17 performs motion compensation on the foreground image of past frames using the motion vector for the foreground image. The super-resolution processing unit 17 inputs the motion-compensated foreground image of past frames along with the foreground image of the current frame into the inference model and infers a high-resolution foreground image (super-resolution processing of the foreground image).

[0082] Super-resolution processing of background images is performed in a similar manner. For example, the super-resolution processing unit 17 acquires motion vectors indicating the movement of the background (BG) as motion vectors for the background image. The super-resolution processing unit 17 uses the motion vectors for the background image to perform motion compensation on the background image of past frames. The super-resolution processing unit 17 inputs the motion-compensated background image of past frames along with the background image of the current frame into the inference model and infers a high-resolution background image (super-resolution processing of background images).

[0083] The super-resolution processing unit 17 combines the super-resolution processed foreground image and background image. The super-resolution processing unit 17 acquires the resulting image as a super-resolution processed image. The super-resolution processing unit 17 outputs the super-resolution processed image to the super-resolution processed image storage unit 18. In this method, the foreground FG and background BG, which are determined to have different movements, are separated and each is processed independently using super-resolution. This prevents the erroneous mixing of two images with significantly different movements (foreground image and background image) by MFSR.

[0084] Furthermore, applying super-resolution processing to the entire screen doubles the processing time. Therefore, it is desirable to separate and process the foreground (FG) and background (BG) only in the vicinity of areas where it is determined that the foreground (FG) and background (BG), which have different movements, are mixed.

[0085] Figure 12 shows an example of generating an image focused on balloon BL while changing the angle of the virtual camera (viewpoint VP).

[0086] In the example in Figure 12, the tree TR in the background is reflected through the translucent balloon BL. The tree TR, which forms the background BG, hardly moves, while only the translucent balloon BL, which forms the foreground FG, moves in response to the wind or changes in the angle of the virtual camera. In the example in Figure 12, the motion vector of the tree TR is (0, 5), and the motion vector of the balloon BL is (50, 100).

[0087] In conventional methods, the motion vector at the superposition of balloon BL and tree TR is calculated as a weighted sum of the motion vectors of balloon BL and tree TR. If the weights of each motion vector are equal, the motion vector at the superposition becomes (25, 52, 5). Applying this motion vector to balloon BL and tree TR at the superposition results in inaccurate motion calculations for both balloon BL and tree TR.

[0088] Therefore, in this disclosure, if there is a significant difference in the movement of the balloon BL and the tree TR, different motion vectors corresponding to their respective movements are assigned to the balloon BL and the tree TR at the superimposed position. This allows for appropriate motion compensation for both the balloon BL and the tree TR, resulting in a super-resolution processed image with minimal blur.

[0089] [5. Hardware Configuration Example] Figure 13 shows an example of the hardware configuration of the information processing device 1.

[0090] The information processing of the information processing device 1 is realized, for example, by a computer 1000. The computer 1000 has a CPU (Central Processing Unit) 1100, RAM (Random Access Memory) 1200, ROM (Read Only Memory) 1300, HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The various parts of the computer 1000 are connected by a bus 1050.

[0091] The CPU 1100 operates based on programs (program data 1450) stored in the ROM 1300 or HDD 1400, and controls each part. For example, the CPU 1100 loads the programs stored in the ROM 1300 or HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0092] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) that are executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0093] The HDD 1400 is a computer-readable non-temporary recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that records an information processing program according to the embodiment, which is an example of program data 1450.

[0094] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (for example, the Internet). For example, the CPU 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0095] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard or mouse via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as a display device, speaker, or printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Discs), tape media, magnetic recording media, or semiconductor memory.

[0096] For example, when computer 1000 functions as an information processing device 1 according to the embodiment, the CPU 1100 of computer 1000 realizes the functions of each part described above by executing an information processing program loaded on RAM 1200. The HDD 1400 stores the information processing program, various models, and various data according to this disclosure. The CPU 1100 reads and executes the program data 1450 from HDD 1400, but as another example, these programs may be obtained from other devices via an external network 1550.

[0097] [6. Effects] The information processing device 1 includes a foreground-background separation image generation unit 19 and a super-resolution processing unit 17. The foreground-background separation image generation unit 19 acquires images of the foreground FG and background BG that overlap in the line of sight as a foreground image and a background image, respectively. The super-resolution processing unit 17 acquires motion vectors for motion compensation of the foreground image and motion vectors for motion compensation of the background image, respectively, based on the motion in the foreground FG and the motion in the background BG that is independent of the foreground FG. In the information processing method of this disclosure, the processing of the information processing device 1 is executed by a computer 1000. The computer-readable non-temporary storage medium of this disclosure stores a program that enables the computer 1000 to implement the processing of the information processing device 1.

[0098] In this configuration, separate motion vectors are assigned to the foreground and background images. Therefore, appropriate motion compensation is performed.

[0099] The foreground section FG allows at least a portion of the background section BG to be seen through to the foreground along the line of sight.

[0100] In this configuration, two movements occur at the perspective position, corresponding to the foreground (FG) and background (BG). Since motion vectors are acquired independently for the foreground and background images, the motion vector of each image does not affect the motion compensation of the other images.

[0101] The foreground section FG, due to its transparency or light-transmitting structure, or due to its movement, allows at least a portion of the background section BG to be seen through in the line of sight.

[0102] With this configuration, even if two motion vectors are generated on the same line of sight due to optical, structural, or rapid movement of the foreground FG, multiple images are less likely to occur.

[0103] The foreground-background separation image generation unit 19 extracts foreground and background images from intermediate data of the radiance field cached in Multiplane Image (MPI) format.

[0104] This configuration ensures that the foreground and background images are properly extracted.

[0105] The foreground-background separation image generation unit 19 calculates the motion vector of each sampling point in the radiance field. Based on the discontinuous distribution of the motion vectors, the foreground FG and background BG to be separated are identified by the foreground FG and background BG.

[0106] With this configuration, the foreground (FG) and background (BG) are properly distinguished.

[0107] The foreground-background separation image generation unit 19 separates and generates the foreground image and background image from the MPI when the movement of the foreground part FG indicated by the motion vector for the foreground image and the movement of the background part BG indicated by the motion vector for the background image differ significantly beyond an acceptable standard.

[0108] With this configuration, the foreground image and background image are appropriately separated based on the independent movement of the foreground (FG) and background (BG).

[0109] The information processing device 1 has a novel viewpoint inference unit 15. The novel viewpoint inference unit 15 infers an image viewed from the direction of the line of sight by superimposing multiple layers PI that constitute the MPI when the difference between the movement of the foreground part FG and the movement of the background part BG satisfies an acceptable standard.

[0110] With this configuration, if the movements of the foreground (FG) and background (BG) are not significantly different, no special processing is performed for foreground-background separation. Foreground-background separation is only performed when there is a large difference in the movements of the foreground (FG) and background (BG), thus reducing the processing load.

[0111] The super-resolution processing unit 17 uses motion vectors for the foreground image and motion vectors for the background image to perform super-resolution processing on the foreground image and background image separately. The super-resolution processing unit 17 then combines the super-resolution processed foreground image and background image to obtain an image viewed from the direction of the line of sight.

[0112] This configuration yields a super-resolution processed image with reduced multiple images.

[0113] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0114] [Note] The technology may also be configured as follows: (1) An information processing device comprising: a foreground-background separation image generation unit that acquires images of a foreground and background that overlap in the line of sight as a foreground image and a background image, respectively; and a super-resolution processing unit that acquires motion vectors for motion compensation of the foreground image and motion vectors for motion compensation of the background image, respectively, based on the movement in the foreground and the movement in the background independent of the foreground. (2) The information processing device according to (1) above, wherein the foreground allows at least a part of the background to be seen through to the foreground side along the line of sight. (3) The information processing device according to (2) above, wherein the foreground allows at least a part of the background to be seen through to the line of sight due to the transparency or light-transmitting structure of the foreground or due to the movement of the foreground. (4) The foreground-background separated image generation unit extracts the foreground image and the background image from intermediate data of the radiance field cached in Multiplane Image (MPI) format, as described in any one of (1) to (3) above. (5) The foreground-background separated image generation unit calculates motion vectors for each sampling point of the radiance field and identifies the foreground and background parts to be separated based on the discontinuous distribution of the motion vectors, as described in (4) above. (6) The foreground-background separated image generation unit separates and generates the foreground image and the background image from the MPI when the movement of the foreground part indicated by the motion vector for the foreground image and the movement of the background part indicated by the motion vector for the background image differ significantly from each other by an acceptable standard. (7) The information processing apparatus according to (6) above, which has a novel viewpoint inference unit that infers an image viewed from the line of sight by superimposing a plurality of layers constituting the MPI when the difference between the movement of the foreground and the movement of the background satisfies the tolerance standard.(8) The information processing apparatus according to any one of (1) to (7) above, wherein the super-resolution processing unit separately performs super-resolution processing on the foreground image and the background image using the motion vector for the foreground image and the motion vector for the background image, and combines the super-resolution processed foreground image and the background image to obtain an image viewed from the direction of the line of sight. (9) An information processing method performed by a computer, comprising: obtaining images of the foreground and background parts that overlap in the direction of the line of sight as a foreground image and a background image, respectively, and obtaining a motion vector for motion compensation of the foreground image and a motion vector for motion compensation of the background image, respectively, based on the movement in the foreground part and the movement in the background part which is independent of the foreground part. (10) A computer-readable non-temporary storage medium storing a program that causes a computer to acquire images of a foreground and background that overlap in the line of sight as a foreground image and a background image, respectively, and to acquire motion vectors for motion compensation of the foreground image and motion compensation of the background image, respectively, based on the movement in the foreground and the movement in the background independent of the foreground.

[0115] 1 Information Processing Unit 15 Novel Viewpoint Inference Unit 17 Super Resolution Processing Unit 19 Foreground / Background Separation Image Generation Unit BG Background Unit FG Foreground Unit

Claims

1. An information processing device comprising: a foreground-background separation image generation unit that acquires images of a foreground and background that overlap in the line of sight as a foreground image and a background image, respectively; and a super-resolution processing unit that acquires motion vectors for motion compensation of the foreground image and motion vectors for motion compensation of the background image, respectively, based on the movement in the foreground and the movement in the background that is independent of the foreground.

2. The information processing apparatus according to claim 1, wherein the foreground portion allows at least a part of the background portion to be seen through to the foreground portion along the line of sight.

3. The information processing apparatus according to claim 2, wherein the foreground portion allows at least a portion of the background portion to be seen through in the line of sight due to the transparency or light-transmitting structure of the foreground portion, or due to the movement of the foreground portion.

4. The information processing apparatus according to claim 1, wherein the foreground-background separation image generation unit extracts the foreground image and the background image from intermediate data of the radiance field cached in Multiplane Image (MPI) format.

5. The information processing apparatus according to claim 4, wherein the foreground-background separation image generation unit calculates motion vectors for each sampling point of the radiance field and identifies the foreground and background parts to be separated based on the discontinuous distribution of the motion vectors.

6. The information processing apparatus according to claim 5, wherein the foreground-background separation image generation unit separates and generates the foreground image and the background image from the MPI when the movement of the foreground part indicated by the motion vector for the foreground image and the movement of the background part indicated by the motion vector for the background image differ significantly by more than an acceptable standard.

7. The information processing apparatus according to claim 6, further comprising a novel viewpoint inference unit that infers an image viewed from the line of sight by superimposing a plurality of layers constituting the MPI when the difference between the movement of the foreground and the movement of the background satisfies the tolerance standard.

8. The information processing apparatus according to claim 1, wherein the super-resolution processing unit separately performs super-resolution processing on the foreground image and the background image using the motion vector for the foreground image and the motion vector for the background image, and combines the super-resolution processed foreground image and background image to obtain an image viewed from the line of sight.

9. A computer-based information processing method comprising: acquiring images of a foreground and background that overlap in the line of sight as a foreground image and a background image, respectively; and acquiring motion vectors for motion compensation of the foreground image and motion compensation of the background image, respectively, based on the movement in the foreground and the movement in the background independent of the foreground.

10. A computer-readable non-temporary storage medium storing a program that causes a computer to acquire images of a foreground and background that overlap in the line of sight as a foreground image and a background image, respectively, and to acquire motion vectors for motion compensation of the foreground image and motion compensation of the background image, respectively, based on the movement in the foreground and the movement in the background independent of the foreground.

Images