Combining intermediate views between wide baseline panoramas

Abstract

1. A method comprising: predicting a stereo depth associated with a first panoramic image and a second panoramic image captured at a time interval; generating a first mesh representation based on the first panoramic image and the corresponding stereo depth; generating a second mesh representation based on the second panoramic image and the corresponding stereo depth; and synthesizing a third panoramic image based on fusing the first mesh representation with the second mesh representation.

Description

[Technical Field]

[0001] field The embodiment relates to panoramic image synthesis. [Background technology]

[0002] background Image stitching, panoramic image stitching, view stitching, frame stitching, and / or similar methods may include generating an image based on at least one existing image and / or frame. For example, frame stitching may include increasing the frame rate of a video by stitching one or more frames between two consecutive adjacent frames. [Overview of the Initiative]

[0003] overview In a general embodiment, a device, system, non-temporary computer-readable medium (storing computer-executable program code that can be run on a computer system), and / or method can perform a process, the method including: predicting stereo depths associated with a first panoramic image and a second panoramic image captured at time intervals; generating a first mesh representation based on the first panoramic image and the stereo depths corresponding to the first panoramic image; generating a second mesh representation based on a second panoramic image and the stereo depths corresponding to the second panoramic image; and synthesizing a third panoramic image based on merging the first mesh representation with the second mesh representation.

[0004] The implementation may include one or more of the following features. For example, the first and second panoramic images may be 360-degree, wide baseline equirectangular projection (ERP) panoramas. The step of predicting stereo depth may estimate the depth of each of the first and second panoramic images using a spherical sweep cost volume based on the first and second panoramic images and at least one target position. The step of predicting stereo depth may estimate a low-resolution depth based on a first feature map associated with the first panoramic image and the second panoramic image, and the step of predicting stereo depth may estimate a high-resolution depth based on a first feature map and a second feature map associated with the first panoramic image. The step of generating a first mesh representation may be obtained based on the first panoramic image and discontinuities determined based on the stereo depth corresponding to the first panoramic image, and the step of generating a second mesh representation may be obtained based on the second panoramic image and discontinuities determined based on the stereo depth corresponding to the second panoramic image.

[0005] The step of generating a first mesh representation may include rendering the first mesh representation into a first 360-degree panorama based on a first target position, and the step of generating a second mesh representation may include rendering the second mesh representation into a first 360-degree panorama based on a second target position, where the first and second target positions may be based on the time interval between the capture of the first panoramic image and the capture of the second panoramic image. The step of synthesizing a third panoramic image may include merging the first mesh representation with the second mesh representation, resolving ambiguity between the first and second mesh representations, and inpainting holes in the synthesized third panoramic image. The step of synthesizing a third panoramic image may include generating a binary visibility mask to identify holes in the first mesh representation based on negative regions in the stereo depth corresponding to the first panoramic image, and to identify holes in the second mesh representation based on negative regions in the stereo depth corresponding to the second panoramic image. The step of synthesizing a third panoramic image may include using a trained neural network, which can join the left and right edges of the third panoramic image using cyclic padding in each convolutional layer.

[0006] The exemplary embodiments should be better understood from the detailed description and accompanying drawings set forth below in this specification, in which similar elements are represented by similar reference figures, which are shown for illustrative purposes only and are not limiting to the exemplary embodiments. [Brief explanation of the drawing]

[0007] [Figure 1A] This diagram shows the acquisition sequence for panoramic images. [Figure 1B] This is a diagram showing a portion of a 360-degree image based on a captured panoramic image. [Figure 1C]This is a block diagram of the panoramic image synthesis process according to one exemplary embodiment. [Figure 2] This is a block diagram of the panoramic image synthesis process according to one exemplary embodiment. [Figure 3] This is a block diagram of a flow for predicting depth, according to one exemplary embodiment. [Figure 4A] This is a block diagram of the flow for training a model to predict depth, according to one exemplary embodiment. [Figure 4B] This is a block diagram of a flowchart for training a model for panoramic image fusion according to one exemplary embodiment. [Figure 5] This is a block diagram of a method for generating a panoramic image sequence according to one exemplary embodiment. [Figure 6] This is a block diagram of a method for synthesizing panoramic images according to one exemplary embodiment. [Figure 7] This is a block diagram of a method for predicting depth according to one exemplary embodiment. [Figure 8] This is a block diagram of a method for training a model to predict depth, according to one exemplary embodiment. [Figure 9] This is a block diagram of a method for training a model for panoramic image fusion according to one exemplary embodiment. [Figure 10] This is a block diagram of a computing system according to at least one exemplary embodiment. [Figure 11] This figure shows an example of a computer device and a mobile computer device according to at least one exemplary embodiment. [Modes for carrying out the invention]

[0008] It should be noted that these figures are intended to illustrate the general characteristics of the methods, structures, and / or materials used in a particular exemplary embodiment, and to supplement the description below. However, these drawings are not to scale and may not accurately reflect the exact structural or performance characteristics of any given embodiment, and should not be construed as defining or limiting the range of values ​​or characteristics encompassed by the exemplary embodiment. For example, the relative thickness and position of elements, layers, regions, and / or structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numerals in various drawings is intended to indicate the presence of similar or identical elements or features.

[0009] Detailed explanation Recent advances in 360-degree cameras and displays (e.g., virtual reality headsets) capable of displaying 360-degree images, image sequences, videos, and / or similar content are increasing the interest of tourists, tenants, photographers, and / or similar individuals in capturing or exploring 360-degree images on computing platforms. These platforms may allow users to virtually walk through cities, view floor plans, and / or similar content (e.g., indoor and outdoor environments) by interpolating between panoramas.

[0010] However, existing solutions lack visual continuity from one view to the next (e.g., from the first panoramic image to the second panoramic image), and ghosting artifacts occur due to warping caused by inaccurate geometry. Existing systems for view synthesis of viewpoint images, single images, and stereoscopic panoramic pairs use a narrow baseline for synthesis.

[0011] In addition, a wide baseline panorama may be used to capture and stream a sequence of panoramic images. Wide baseline images (including wide baseline panoramas) are images that involve a relatively large amount of camera motion (e.g., distance, rotation, translation, and / or the like) and changes in the (camera's) internal parameters between two views (e.g., from the first panoramic image to the second panoramic image). For example, in the frames of a movie, the changes in camera motion and internal parameters may be relatively small between the first and second frames of the video. However, the changes in camera motion and internal parameters may be relatively large between, for example, the first and tenth frames, the first and one-hundredth frames, the first and one-thousandth frames (e.g., wide baseline) of the video.

[0012] Existing systems have limitations when processing wide baseline panoramas because they do not include the synthesis of omnidirectional videos with large motion (e.g., using a pair of wide baseline panoramas). Therefore, existing platforms may not be configured to perform view synthesis of wide baseline panoramas.

[0013] Exemplary implementations can generate a video by synthesizing a wide baseline panorama to fill the visual gaps between panorama images in a sequence of panorama images. The resulting video can be streamed as a 360-degree video to a computing device (e.g., an augmented reality (AR) device) for an interactive and seamless user experience. Alternatively, exemplary implementations can stream a wide baseline panorama to a consumer device configured to synthesize a 360-degree video between wide baseline panoramas and display the resulting 360-degree video on the consumer device for an interactive and seamless experience. Different from existing systems that only synthesize new views inside a limited volume or along a trajectory in a linear projection, exemplary implementations can generate a 360-degree video, thereby enabling (or assisting in enabling) the user to move forward / backward, stop anywhere, and look around from any viewpoint. This enables a wide range of applications such as cinematography, video conferencing, and virtual tourism trips, and / or the like (e.g., virtual reality applications). Thus, by synthesizing views of wide baseline panoramas, the functionality of the platform can be improved, and the user can be enabled to virtually walk through a city and view test shots of floor plans and / or the like (e.g., indoor and outdoor environments). Synthesizing views of wide baseline panoramas can enable a full field of view (e.g., a 360-degree field of view) by enabling alignment between two panoramas.

[0014] Figure 1A shows the acquisition sequence for panoramic images. As shown in Figure 1A, multiple panoramas 10-1, 10-2, 10-3, 10-4, ..., 10-n (e.g., a wide baseline panorama or wide baseline panoramic image) can be acquired as images in an image sequence. After the panoramic images have been acquired, there may be acquisition intervals 20-1, 20-2, 20-3, 20-4, ..., 20-n. The acquisition intervals 20-1, 20-2, 20-3, 20-4, ..., 20-n (or acquisition time intervals) can be caused by the time when the camera (e.g., a 360-degree camera) is not acquiring images. In other words, since the camera is not acquiring data continuously (like video), it can acquire sequences of images that are not capturing video. Therefore, there are periods (e.g., time and distance) between the image acquisitions shown as acquisition intervals 20-1, 20-2, 20-3, 20-4, and 20-n where delays occur. In some implementations, the capture intervals 20-1, 20-2, 20-3, 20-4, ..., 20-n may result in distance gaps of at least 5 m corresponding to the capture intervals. The results of illustrating the capture intervals 20-1, 20-2, 20-3, 20-4, ..., 20-n can be shown in Figure 1B.

[0015] Figure 1B shows a portion of a 360-degree image based on an captured panoramic image. As shown in Figure 1B, multiple panoramas 30-1, 30-2, 30-3, 30-4, 30-5, 30-6, 30-7, 30-8, 30-9 (e.g., wide baseline panoramas or wide baseline panoramic images) may be used to generate a portion of the 360-degree image. The portion of the 360-degree image may be generated based on a 3D position (e.g., x, y, z) within a range of corresponding locations (e.g., geographical location, spatial, and / or similar), using, for example, the Global Positioning System (GPS), location anchors, and / or similar. As shown in Figure 1B, there may be gaps 40-1, 40-2 (e.g., distance) between two or more panoramas 30-1, 30-2, 30-3. The gaps 40-1 and 40-2 can be based on the capture intervals 20-1, 20-2, 20-3, 20-4, ..., 20-n. While gaps 40-1 and 40-2 are shown as smaller than panoramas 30-1, 30-2, and 30-3, they can be smaller or larger than panoramas 30-1, 30-2, and 30-3, or even the same size. In other words, gaps 40-1 and 40-2 can be any size relative to panoramas 30-1, 30-2, and 30-3. While gaps 40-1 and 40-2 are shown in a horizontal sequence (e.g., horizontal direction), they can also be in a vertical sequence and / or a diagonal sequence. Gaps 40-1 and 40-2 may impair the user experience when viewing 360-degree video. Therefore, as outlined with respect to Figure 1C, the exemplary implementation may include techniques used to shorten or eliminate gaps 40-1, 40-2, 50-1, 50-2 that may be caused by the intake intervals 20-1, 20-2, 20-3, 20-4, ..., 20-n.

[0016] Figure 1C shows a block diagram of a panoramic image stitching flow according to one exemplary embodiment. As shown in Figure 1C, the image stitching flow 100 includes n panoramas 105, depth prediction 110 blocks, differential rendering 115 blocks, fusion 120 blocks, and a stitched panorama 125.

[0017] n panoramas 105 can be a sequence of n panoramic images captured by a rotating camera. Each of the n panoramas 105 can be a partial (e.g., 180-degree) two-dimensional (2D) projection of a three-dimensional (3D) view, captured using a 360-degree rotation (e.g., camera rotation).

[0018] The depth prediction block 110 may be configured to predict the depth associated with each of the n panoramas 105. The depth may be based on two adjacent panoramas in a sequence of n panoramas 105. The differential rendering block 115 may be configured to generate an RGB panorama and / or an RGBD panorama based on the depth prediction and viewpoint corresponding to the target location. The target location may be a differential location based on the location associated with the panorama. The target location may be associated with one or more of the gaps 40-1, 40-2, 50-1, 50-2 that can be brought about by the acquisition intervals 20-1, 20-2, 20-3, 20-4, ..., 20-n.

[0019] The fusion block 120 may be configured to generate a composite panorama 125 based on at least two differentially rendered panoramas. The composite panorama 125 may be inserted between two of the n panoramas 105 into a sequence of images containing n panoramas 105. A more detailed explanation for generating the composite panorama is described in relation to Figure 2.

[0020] Figure 2 shows a block diagram of a panoramic image synthesis flow according to one exemplary embodiment. As shown in Figure 2, the panoramic image synthesis flow 200 includes panoramas 205, 210, depth predictors 215, 220, depth prediction 225, 230 blocks, differential mesh renderers 235, 240, target position 245, 250 blocks, RGB 255-1, 260-1 blocks, visibility 255-2, 260-2 blocks, fusion network 265, and synthesized panorama 270.

[0021] Panoramas 205 and 210 may be images captured by a rotating camera. Panoramas 205 and 210 may be captured using a fisheye lens. Therefore, panoramas 205 and 210 may be a partial (e.g., 180-degree) 2D projection of a 3D view, captured using 360-degree rotation (e.g., camera rotation). Panoramas 205 and 210 may include global and local alignment information. Global and local alignment information may include position (e.g., coordinates), displacement, orientation information, pitch, roll, yaw (e.g., position relative to the x, y, and z axes), and / or other information used to align two or more panoramas. Position may be from a Global Positioning System (GPS), a position anchor (e.g., in a room), and / or similar. Panoramas 205 and 210 may be panoramas with a wide baseline. In panoramas with a wide baseline, the acquisition characteristics of two or more images may vary significantly. In the exemplary implementation, the significant variation may be based on the position of the acquisition camera. In other words, the camera is moving at a speed that creates gaps between the images. Panoramas 205, 210 can be stored (or received, input, and / or similar) as a mesh.

[0022] Depth predictors 215, 220 may be configured to determine the depth associated with each pixel of panoramas 205, 210. As shown, depth predictors 215, 220 can determine the depth using both panoramas 205 and 210. Depth predictors 215, 220 can determine the depth of each panorama 205, 210 using a machine learning model. Depth predictors 215, 220 can generate depth predictions 225, 230. Depth predictions 225, 230 may be stereo depth estimations using monocular connections (one or more). Stereo depth estimation can allow for the matching of features presented in two or more 360-degree images (e.g., panoramas 205, 210) for aligned depth estimation. Monocular connections (one or more) can allow for depth predictions of regions confined in the first image, which may or may not be confined in the second image. Depth predictors 215 and 220 are described in more detail below.

[0023] The differential mesh renderers 235, 240 may be configured to generate RGB 255-1, 260-1 and visibility 255-2, 260-2 based on depth prediction 225, 230 and target positions 245, 250. Each image may be rendered from a viewpoint corresponding to the target positions 245, 250. The target positions 245, 250 may be differential positions based on positions related to panoramas 205, 210. The target positions 245, 250 may be associated with one or more gaps (e.g., gaps 40-1, 40-2, 50-1, 50-2) in a sequence of images that may be brought about by an image acquisition interval (or acquisition time interval) (e.g., acquisition intervals 20-1, 20-2, 20-3, 20-4, ..., 20-n). The differential mesh renderers 235, 240 may be configured to generate a spherical mesh for each of the panoramas 205, 210. Instead of a point cloud representation, a mesh representation of Panorama 205 or 210 may be used, as this can avoid density problems associated with generating point clouds from ERP images. For example, point clouds generated from ERP images may contain large changes in sparseness and density when traveling long distances, which can make inpainting (e.g., filling in holes in arbitrary topology so that the additions appear to be part of the original image) difficult.

[0024] For output images with W×H resolution, the difference mesh renderers 235, 240 may be configured to generate a spherical mesh that follows a UV pattern having a height segment of 2H and a width segment of 2W. The vertices can then be offset to an exact radius based on the Euclidean depth d from the depth predictions 225, 230. After generating the mesh and offsetting the vertices to their exact depths, the difference mesh renderers 235, 240 may be configured to calculate the gradients of the depth map along the θ and φ directions to yield gradient images dθ and dφ. These gradient images can represent the estimation of the normals of each surface. Larger gradients in the depth images correspond to the edges of buildings and other structures within the RGB image. These surfaces may have normal vectors perpendicular to the vector from the camera position. The difference mesh renderers 235, 240 may be configured to set thresholds for the depth gradient along both directions to identify discontinuities in the 3D structure where (dθ>k)|(dφ>k). For these regions, the differential mesh renderers 235 and 240 may be configured to reject triangles inside the spherical mesh in order to accurately represent the inherent discontinuities.

[0025] Once the mesh is generated and discontinuities are calculated, the differential mesh renderers 235, 240 may be configured to render the mesh from a new viewpoint to RGB255-1, 260-1 (for example, a 360-degree RGBD image). The mesh rendering may contain holes due to occlusion in the original image. These holes may be represented as negative values ​​in the depth image. The differential mesh renderers 235, 240 may be configured to extract visibility255-2, 260-2 from the negative values.

[0026] In the example implementation, the differential mesh renderers 235 and 240 may be configured to adapt a mesh renderer (e.g., an internal mesh renderer) to output a 360-degree image. For example, the rasterizer may be modified to project vertices from world coordinates to camera coordinates and then to screen coordinates. The differential mesh renderers 235 and 240 may be configured to normalize the final coordinates to, for example, [-1;1] by applying a transformation from Cartesian to spherical coordinates, rather than multiplying the camera coordinates of the vertices by a projection matrix.

[0027] In the example implementation, the difference mesh renderers 235 and 240 may be configured to perform a normal rendering pass and a rendering pass that rotates 180 degrees, and then combine the passes so that the triangles wrapping around the left and right edges of the panorama are not lost in the final rendering. In addition, the difference mesh renderers 235 and 240 may be configured to use a dense mesh to minimize the length of each triangle in the final image. Performing two rendering passes and using a dense mesh can minimize (or prevent) the loss of triangles wrapping around the left and right edges of panoramas 205 and 210, which would result in the inaccurate mapping of rectangular coordinate lines to coordinate lines in the ERP image. The steps of performing two rendering passes and using a dense mesh can be performed simultaneously by rendering six viewpoint planes of a cubemap and projecting that cubemap onto an equirectangular projection image.

[0028] The fusion network 265 may be configured to generate a composite panorama 270. The fusion network 265 may be configured to fuse RGB260-1 with RGB255-1. RGB255-1,260-1 may contain holes due to occlusion in the composite view (for example, RGB255-1,260-1 are composited at target positions 245,250). Therefore, the fusion network 265 may be configured to inpaint the holes.

[0029] The fused network 265 may be configured to generate a synthetic panorama 270 (e.g., a single invariant panorama) using a trained model (e.g., a trained neural network). The trained neural network may include seven downsampling elements and seven upsampling elements. In one exemplary implementation, the fused network 265 may be configured to generate a binary visibility mask to identify holes (e.g., negative regions in the mesh rendering the depth image) in RGB255-1, 260-1, respectively, based on visibility 255-2, 260-2. The fused network 265 may be configured to use cyclic padding in each convolutional layer to simulate a cyclic convolutional neural network (CNN) and join the left and right edges. Zero padding may be used at the beginning and end of each feature map.

[0030] The aforementioned depth pipeline can use a neural network (e.g., a CNN) with five downsampling blocks and three upsampling blocks as a feature encoder, a 3D neural network (e.g., a CNN) with three downsampling blocks and three upsampling blocks as a cost-volume improvement network, and two convolutional blocks as depth decoders. The depth pipeline can use vertical input indices as additional channels for each convolutional layer. This allows the convolutional layers to learn distortions associated with equirectangular projection (ERP). The depth pipeline will be discussed in more detail with respect to Figure 3.

[0031] Figure 3 shows a block diagram of a flow for depth prediction according to one exemplary embodiment. As shown in Figure 3, the depth prediction flow 300 (related to depth predictors 215, 220, for example) includes panorama blocks 305, 310, 2D convolution blocks 315, 320, 350, 360, feature map blocks 325, 330, 345, cost volume block 335, 3D convolution block 340, and depth blocks 355, 365.

[0032] Panoramas 305 and 310 may be images captured by a rotating camera. Panoramas 305 and 310 may be captured using a fisheye lens. Therefore, panoramas 305 and 310 may be a partial (e.g., 180-degree) 2D projection of a 3D view, captured using 360-degree rotation (e.g., camera rotation). Panoramas 305 and 310 may include global and local alignment information. Global and local alignment information may include position (e.g., coordinates), displacement, orientation information, pitch, roll, yaw (e.g., position relative to the x, y, and z axes), and / or other information used to align two or more panoramas. Position may be from a Global Positioning System (GPS), a position anchor (e.g., in a room), and / or similar. Panoramas 305 and 310 may be panoramas with a wide baseline. In panoramas with a wide baseline, the acquisition characteristics of two or more images may vary significantly. In the exemplary implementation, the significant variation may be based on the position of the acquisition camera. In other words, the camera is moving at a speed that creates gaps between the images. Panoramas 305 and 310 can be stored (or received, input, and / or similar) as a mesh.

[0033] The 2D convolutional blocks 315 and 320 may be configured to generate features related to the panoramas 305 and 310. The 2D convolutional blocks 315 and 320 may be a trained neural network (e.g., a CNN). The 2D convolutional blocks 315 and 320 may be a reduction path (e.g., an encoder) related to the convolutional model (the 2D convolutional blocks 350 and 360 are expansion paths (e.g., decoders)). The 2D convolutional blocks 315 and 320 may be a classification network (e.g., VGG / ResNet) to which downsampling by max pooling is applied following the convolutional blocks to encode the panoramas 305 and 310 into feature representations at multiple different levels. These multiple different levels of feature representations may be feature maps 325 and 330.

[0034] The cost volume 335 block may be configured to generate a spherical sweep cost volume of features based on feature maps 325, 330. The cost volume may be a measure of similarity between all pairs of reference points and matching candidate points in feature maps 325, 330. The spherical sweep may be configured to align feature map 325 to feature map 330. The spherical sweep may include the step of transforming feature maps 325, 330 into a spherical region. The step of transforming feature maps 325, 330 may include the step of projecting feature maps 325, 330 onto a given sphere. The step of generating a spherical sweep cost volume of features may include integrating the feature maps 325, 330 with the associated spherical volumes, and for stereo matching (e.g., matching a patch from panorama 305 centered at position p with a patch from panorama 310 centered at position pd), the cost is calculated using the integrated spherical volumes as input to a cost function (e.g., sum of absolute differences (SAD), sum of squared differences (SSD), normalized cross-correlation (NCC), zero-mean-based cost (such as ZSAD, ZSSD, and ZNCC)) based on a first image derivative (gradient) or a second image derivative (Gaussian Laplacian) and / or similar.

[0035] A 3D convolution 340 block may be configured to improve the cost volume. The cost volume improvement step may include aggregating feature information along mismatch dimensions or spatial dimensions (one or more). The 3D convolution 340 may be a 3D neural network (e.g., a CNN). The 3D neural network may include three downsampling blocks and three upsampling blocks as a cost volume improvement network. Improving the cost volume can generate a feature map. The feature map may be a feature map 345.

[0036] Feature map 345 can be input to 2D convolutional blocks 350 and 2D convolutional blocks 360. 2D convolutional blocks 350 and 360 can be used as depth decoders (e.g., depth predictors) to generate (e.g., predict) depth blocks 355 and 365. Depth decoding may involve a step using two convolutional blocks. Feature map 345 can be input to 2D convolutional blocks 360. Feature map 325 can be used as a vertical input index as an additional channel for each convolutional layer in the depth prediction network. This allows the convolutional layers to learn distortions associated with equirectangular projection (ERP). Depth predictions, as described with respect to Figure 3, can be trained. For example, depth predictions can be associated with depth predictors 215 and 220. Training of the neural network associated with depth predictions is described with respect to Figure 4A.

[0037] Figure 4A shows a block diagram of the flow for training a model to predict depth according to one exemplary embodiment. As shown in Figure 4A, training a model to predict depth includes panorama blocks 205, 210, depth predictor blocks 215, 220, depth prediction blocks 225, 230, loss block 410, and training block 420.

[0038] The depth predictor 215 uses two panoramas 205, 210 (e.g., wide baseline images in a sequence) as input for training. The depth predictor 215 includes two outputs (e.g., depth 355 and depth 365), the first output (e.g., depth 355) is a low-resolution depth d based only on cost volume (e.g., cost volume 335). pred_low The second output (e.g., depth 365) includes the prediction, and the second output (e.g., depth 365) is a high-resolution depth d from the feature map (e.g., feature map 325) and cost volume (e.g., cost volume 335). pred_hi This includes predictions. The first output can be associated with gradient flow. In one exemplary implementation, the depth loss function associated with loss 410 blocks may be:

[0039]

number

[0040] Here, l depth This is the depth loss, d gt This is the threshold for the depth gradient, λ is the scaling factor (for example, λ=0.5), d pred_hi This is high-resolution depth, d pred_low This is a low-resolution depth.

[0041] The training block 420 may be configured to result in training of the depth predictor 215. In one exemplary implementation, the depth predictor 215 includes 2D convolution blocks 315, 320, 350, 360 and a 3D convolution block 340, each with weights associated with the convolution. Training the depth predictor 215 may include a step of changing these weights. Changing the weights may result in changes to two outputs (e.g., depth 355 and depth 365) (even when using the same input panorama). Changes in the two outputs (e.g., depth 355 and depth 365) may affect the depth loss (e.g., loss 410). Training may be repeated until the loss 410 is minimized and / or until there are no significant changes in the loss 410 between iterations.

[0042] Figure 4B shows a block diagram of a flow for training a model for panoramic image fusion according to one exemplary embodiment. As shown in Figure 4B, training a model for panoramic image fusion includes panoramas 430-1, 430-2, 430-3, target position blocks 245, 250, RGB blocks 255-1, 260-1, visibility blocks 255-2, 260-2, fusion network block 265, composite panorama block 270, loss block 440, and training block 450.

[0043] Training the fusion network 265 involves a step using a sequence of three panoramas (panoramas 430-1, 430-2, and 430-3). Using the pose of the intermediate panorama (panorama 430-2), a mesh rendering can be generated from the first panorama (panorama 430-1) and the last panorama (panorama 430-3). The fusion network 265 receives the mesh rendering and can combine the mesh renderings to predict the intermediate panorama (e.g., panorama 270). The intermediate panorama (panorama 430-2), which is the ground truth, is used for supervision. A loss 440 can be used to train the fusion network 265. The loss 440 can be determined by the following equation:

[0044]

number

[0045] Here, l fusion This is the fusion loss (for example, loss 440), p1 is a ground trough panorama (e.g., panorama 430-2), p pred This is the predicted panorama (Panorama 270).

[0046] The training block 450 can be configured to effect the training of the fusion network 265. The training of the fusion network 265 can include the step of changing the weights associated with at least one of the convolutions of the fusion network 265. In one exemplary implementation, the fusion network 265 can be trained based on the difference between the predicted panorama (e.g., panorama 270) and the ground truth panorama (e.g., panorama 430-2). A loss (e.g., loss 440) can be generated based on the difference between the predicted panorama and the ground truth panorama. The training can be repeated until the loss 440 is minimized and / or until there is no significant change in the loss 440 between iterations. In one exemplary implementation, as the loss becomes smaller, the synthesis (e.g., prediction) of the intermediate panorama by the fusion network 265 becomes better. Additionally, when the depth predictor 215 and the fusion network 265 are trained together, the total loss can be l total =l depth +l fusion and can be obtained.

[0047] FIG. 5 shows a block diagram of a method for generating a panorama image sequence according to one exemplary embodiment. As shown in FIG. 5, at step S505, it is determined that there is an image capture interval (or capture time interval) between two or more panorama images in the image sequence. For example, the image sequence or panorama image sequence can be captured by a rotating camera. Each panorama image in the image sequence can be a partial (e.g., 180-degree) 2D projection of a 3D view captured using a 360-degree rotation (e.g., rotation of the camera). The capture interval (or capture time interval) can be brought about by the time during which the camera (e.g., 360-degree camera) is not capturing an image. In other words, since the camera is not continuously capturing data (like a video), it can capture a sequence of images during which it is not capturing a video. Thus, there is a period (e.g., time and distance) during which a delay occurs while capturing an image. In some implementations, the capture interval can potentially result in a distance gap of at least 5 meters between images.

[0048] In step S510, a composite image is generated based on two or more panoramic images. For example, if there is an image acquisition interval (or acquisition time interval), the exemplary implementation can composite at least one panoramic image to be inserted into the sequence of images in order to reduce and / or eliminate the distance gap between two panoramic images. In step S515, the composite image is inserted into the sequence of images between the two or more panoramic images. For example, referring to Figure 1B, a composite image may be inserted to minimize and / or eliminate one or more of the gaps 40-1, 40-2, 50-1, and 50-2.

[0049] Figure 6 shows a block diagram of a method for synthesizing panoramic images according to one exemplary embodiment. As shown in Figure 6, a first panoramic image and a second panoramic image are received in step S605. For example, the panoramas (panoramas 205, 210) may be images captured by a rotating camera. The panoramas may be captured using a fisheye lens. Thus, the panoramas may be a partial (e.g., 180-degree) 2D projection of a 3D view, captured using a 360-degree rotation (e.g., camera rotation). The panoramas may include global alignment information and local alignment information. The global and local alignment information may include position (e.g., coordinates), displacement, orientation information, pitch, roll, yaw (e.g., position relative to the x, y, and z axes), and / or other information used to align two or more panoramas. The position may be from a Global Positioning System (GPS), a position anchor (e.g., in a room), and / or similar. The panoramas may be wide baseline panoramas. In wide baseline panoramas, the acquisition characteristics of two or more images may vary significantly. In the example implementation, significant variations may be based on the position of the acquisition camera. In other words, the camera is moving at a speed that creates gaps between images. The panorama can be stored (or received, input, and / or similar) as a mesh.

[0050] In step S610, a first depth prediction is generated based on the first panoramic image and the second panoramic image. For example, the first depth prediction may include the step of determining the depth associated with each pixel in the first panorama. The first depth prediction may be based on both the first and second panoramas. The first depth prediction can determine the depth(s) of the panoramas using a machine learning model. The depth prediction may be a stereo depth estimation using monocular connection(s). Stereo depth estimation can allow for the matching of features presented in two or more 360-degree images (e.g., panoramas 205, 210) for aligned depth estimation. Monocular connection(s) can allow for depth prediction of regions confined in the first panoramic image, which may or may not be confined in the second panoramic image.

[0051] In step S615, a first differential mesh is generated based on a first depth prediction. For example, a differential mesh renderer (e.g., differential mesh renderer 235) can generate RGB-D images (e.g., RGB255-1 and visibility map (e.g., visibility255-2) based on a first depth prediction (e.g., depth prediction 225) and a target location (e.g., target location 245)). Each image may be rendered from a viewpoint corresponding to the target location. The target location may be a differential location based on a location related to a first panorama and a second panorama. The first differential mesh may be a spherical mesh corresponding to the first panorama. A mesh representation of the first panorama may be used instead of a point cloud representation, because this can avoid density problems associated with generating a point cloud from an ERP image. For example, a point cloud generated from an ERP image may contain large changes in sparseness when traveling large distances, which can make inpainting (e.g., filling in holes in arbitrary topology so that the additions appear to be part of the original image) difficult.

[0052] In step S620, a second depth prediction is generated based on the second panoramic image and the first panoramic image. For example, the second depth prediction may include the step of determining the depth associated with each pixel in the second panorama. The second depth prediction may be based on both the first and second panoramas. The second depth prediction can determine the depth of the panorama(s). The depth prediction may be a stereo depth estimation using monocular connection(s). Stereo depth estimation can enable the matching of features presented in two or more 360-degree images (e.g., panoramas 205, 210) for aligned depth estimation. Monocular connection(s) can enable depth prediction of regions confined in the second panoramic image, which may or may not be confined in the first panoramic image.

[0053] In step S625, a second differential mesh is generated based on a second depth prediction. For example, a differential mesh renderer (e.g., differential mesh renderer 235) can generate RGB-D images (e.g., RGB260-1 and visibility map (e.g., visibility260-2) based on a second depth prediction (e.g., depth prediction 230) and a target location (e.g., target location 250)). Each image may be rendered from a viewpoint corresponding to the target location. The target location may be a differential location based on the locations associated with the first and second panoramas. The first differential mesh may be a spherical mesh corresponding to the second panoramic image. A mesh representation of the second panoramic image may be used instead of a point cloud representation, because this can avoid density problems associated with generating a point cloud from an ERP image. For example, a point cloud generated from an ERP image may contain large changes in sparseness when traveling large distances, which can make inpainting (e.g., filling in holes in arbitrary topology so that the additions appear to be part of the original image) difficult.

[0054] In step S630, a composite panoramic image is generated by fusing a first difference mesh with a second difference mesh. For example, a fusing network (e.g., fusing network 265) can fuse an RGB-D image (e.g., RGB255-1) associated with the first difference mesh with an RGB-D image (RGB260-1) associated with the second difference mesh. The RGB-D image(s) may contain holes due to occlusion in the composite view at target positions 245, 250. Thus, the fusing may include inpainting of holes. The fusing can generate a composite panorama using a trained model (e.g., a trained neural network). The trained neural network may include seven downsampling elements and seven upsampling elements. In one exemplary implementation, the fusing may include a step of identifying holes (e.g., negative regions in the mesh rendering depth image) by generating a binary visibility mask based on visibility maps (e.g., visibility 255-2, 260-2) in each of the RGB-D images. The fusion process may involve a step that simulates a circular convolutional neural network (CNN) by using circular padding in each convolutional layer to join the left and right ends. Zero padding can be used at the beginning and end of each feature map.

[0055] Figure 7 shows a block diagram of a method for predicting depth according to one exemplary embodiment. As shown in Figure 7, a first panoramic image and a second panoramic image are received in step S705. For example, the panoramas (panoramas 205, 210) may be images captured by a rotating camera. The panoramas may be captured using a fisheye lens. Thus, the panoramas may be a partial (e.g., 180-degree) 2D projection of a 3D view, captured using a 360-degree rotation (e.g., camera rotation). The panoramas may include global alignment information and local alignment information. The global and local alignment information may include position (e.g., coordinates), displacement, orientation information, pitch, roll, yaw (e.g., position relative to the x, y, and z axes), and / or other information used to align two or more panoramas. The position may be from a Global Positioning System (GPS), a position anchor (e.g., in a room), and / or similar. The panoramas may be wide baseline panoramas. In wide baseline panoramas, the acquisition characteristics of two or more images may vary significantly. In the example implementation, significant variations may be based on the position of the acquisition camera. In other words, the camera is moving at a speed that creates gaps between images. The panorama can be stored (or received, input, and / or similar) as a mesh.

[0056] In step S710, a first map is generated based on the first panoramic image. For example, a neural network may be used to generate features associated with the first panorama. In one exemplary implementation, the 2D convolution may be a trained neural network (e.g., a CNN). The 2D convolution may be a reduction path associated with the convolutional model (e.g., an encoder). The 2D convolution may be a classification network (e.g., VGG / ResNet) to which downsampling by max pooling is applied following the convolutional block to encode the first panorama into feature representations at multiple different levels. These feature representations at multiple different levels may be the first feature map.

[0057] In step S715, a second feature map is generated based on the second panoramic image. For example, a neural network may be used to generate features associated with the second panorama. In one exemplary implementation, the 2D convolution may be a trained neural network (e.g., a CNN). The 2D convolution may be a reduction path associated with the convolutional model (e.g., an encoder). The 2D convolution may be a classification network (e.g., VGG / ResNet) to which downsampling by max pooling is applied following the convolutional block to encode the second panorama into feature representations at multiple different levels. These feature representations at multiple different levels may be the second feature map.

[0058] In step S720, a cost volume is generated based on the first and second feature maps. For example, a spherical sweep cost volume of features can be determined (or generated) based on the first and second feature maps (e.g., feature maps 325 and 330). The cost volume can be a measure of similarity between all pairs of reference points and matching candidate points in the feature map. The spherical sweep may be configured to align the first feature map with the second feature map. The spherical sweep may include a step of transforming the feature map into a spherical region. The step of transforming the feature map may include a step of projecting the feature map onto a given sphere. The step of generating a spherical sweep cost volume of features may include integrating the feature map and the associated spherical volume, and for stereo matching (e.g., matching a patch from a first panorama centered at position p with a patch from a second panorama centered at position pd), the cost is calculated using the integrated spherical volume as input to a cost function (e.g., sum of absolute differences (SAD), sum of squared differences (SSD), normalized cross-correlation (NCC), zero-mean-based cost (such as ZSAD, ZSSD, and ZNCC)) based on a first image derivative (gradient) or a second image derivative (Gaussian Laplacian) and / or similar.

[0059] In step S725, a third feature map is generated based on the cost volume. For example, the third feature map may be generated by refining the cost volume. The step of refining the cost volume may include a step of aggregating feature information along mismatch dimensions or spatial dimensions (one or more). The step of refining the cost volume may include a step of using a 3D convolutional neural network (e.g., CNN). The 3D neural network may include three downsampling blocks and three upsampling blocks as a cost volume refining network. Refining the cost volume can generate a third feature map.

[0060] In step S730, a first depth is generated based on a third feature map. For example, a 2D convolution may be used as a depth decoder (e.g., depth predictor) to generate (e.g., predict) the first depth. Depth decoding may involve a step using two convolutional blocks. The depth predictor may be a trained depth predictor.

[0061] In step S735, a second depth is generated based on the first and third feature maps. For example, a 2D convolution may be used as a depth decoder (e.g., depth prediction) to generate (e.g., predict) the first depth. Depth decoding may involve a step using two convolutional blocks. The first feature map may be input to the 2D convolution. The first feature map may be used as a vertical input index as an additional channel for each convolutional layer in the depth prediction network. This allows the convolutional layers to learn the distortion associated with the equirectangular projection (ERP).

[0062] Figure 8 shows a block diagram of a method for training a model to predict depth according to one exemplary embodiment. As shown in Figure 8, a first panoramic image and a second panoramic image are received in step S805. For example, the panoramas (panoramas 205, 210) may be images captured by a rotating camera. The panoramas may be captured using a fisheye lens. Thus, the panoramas may be a partial (e.g., 180-degree) 2D projection of a 3D view, captured using a 360-degree rotation (e.g., camera rotation). The panoramas may include global alignment information and local alignment information. The global and local alignment information may include position (e.g., coordinates), displacement, orientation information, pitch, roll, yaw (e.g., position relative to the x, y, and z axes), and / or other information used to align two or more panoramas. The position may be from a Global Positioning System (GPS), a position anchor (e.g., in a room), and / or similar. The panoramas may be panoramas with a wide baseline. In panoramas with a wide baseline, the acquisition characteristics of two or more images can vary significantly. In the exemplary implementation, this significant variation may be based on the position of the acquisition camera. In other words, the camera is moving at a speed that creates gaps between images. The panorama can be stored (or received, input, and / or similar) as a mesh.

[0063] In step S810, a first depth is generated based on the first panoramic image and the second panoramic image. In step S815, a second depth is generated based on the first panoramic image and the second panoramic image. The steps for generating the first and second depths are described above in relation to steps S730 and S735 in Figure 7, for example. For example, depth prediction can use two panoramas (e.g., wide baseline images in a sequence) as input for training. Depth prediction may include two outputs (e.g., depth 355 and depth 365), where the first output (e.g., depth 355) is a low-resolution depth d based only on the cost volume (e.g., cost volume 335). pred_lowThe second output (e.g., depth 365) includes the prediction, and the second output (e.g., depth 365) is a high-resolution depth d from the feature map (e.g., feature map 325) and cost volume (e.g., cost volume 335). pred_hi This includes predictions. The first output can be associated with gradient flow.

[0064] In step S820, the loss is calculated based on the first depth and the second depth. For example, as discussed above, to calculate the loss, the low-resolution depth d pred_low and high-resolution depth d pred_hi A loss function for depth based on the above may be used.

[0065] In step S825, the depth prediction is trained based on the loss. For example, the depth prediction may include the use of at least one 2D convolution and at least one 3D convolution, each having weights associated with the convolution. Training the depth prediction may include a step of changing these weights. Changing the weights may result in changes to two outputs (e.g., depth 355 and depth 365) (even when using the same input panorama). Changes in the two outputs (e.g., depth 355 and depth 365) may affect the depth loss (e.g., loss 410). Training may be repeated until the loss is minimized and / or until there are no significant changes in the loss between iterations.

[0066] Figure 9 shows a block diagram of a method for training a model for panoramic image fusion according to one exemplary embodiment. As shown in Figure 9, a sequence of panoramic images is received in step S905. For example, the panorama (e.g., panoramas 430-1, 430-2, 430-3) may be images captured by a rotating camera. The panorama may be captured using a fisheye lens. Thus, the panorama may be a partial (e.g., 180-degree) 2D projection of a 3D view, captured using a 360-degree rotation (e.g., camera rotation). The panorama may include global alignment information and local alignment information. The global and local alignment information may include position (e.g., coordinates), displacement, orientation information, pitch, roll, yaw (e.g., position relative to the x, y, and z axes), and / or other information used to align two or more panoramas. The position may be from a Global Positioning System (GPS), a position anchor (e.g., in a room), and / or similar. The panorama may be a wide baseline panorama. In panoramas with a wide baseline, the acquisition characteristics of two or more images can vary significantly. In the exemplary implementation, this significant variation may be based on the position of the acquisition camera. In other words, the camera is moving at a speed that creates gaps between images. The panorama can be stored (or received, input, and / or similar) as a mesh.

[0067] In step S910, a first difference mesh is generated based on the first panoramic image of the sequence of panoramic images. For example, a difference mesh renderer (e.g., difference mesh renderer 235) can generate an RGB-D image (e.g., RGB255-1 and a visibility map (e.g., visibility255-2) based on the first panoramic image and depth predictions associated with the target location (e.g., target location 245)). Each image may be rendered from a viewpoint corresponding to the target location. The target location may be a difference location based on the locations associated with the first and second panoramas. The first difference mesh may be a spherical mesh corresponding to the first panoramic image. A mesh representation of the first panoramic image may be used instead of a point cloud representation, because this can avoid density problems associated with generating a point cloud from an ERP image. For example, a point cloud generated from an ERP image may contain large changes in sparseness when traveling large distances, which can make inpainting (e.g., filling in holes in arbitrary topology so that the additions appear to be part of the original image) difficult.

[0068] In step S915, a second difference mesh is generated based on the second panoramic image of the panoramic image sequence. For example, a difference mesh renderer (e.g., difference mesh renderer 240) can generate an RGB-D image (e.g., RGB260-1 and a visibility map (e.g., visibility260-2) based on the second panoramic image and depth predictions associated with the target location (e.g., target location 245)). Each image may be rendered from a viewpoint corresponding to the target location. The target location may be a difference location based on the locations associated with the first and second panoramas. The first difference mesh may be a spherical mesh corresponding to the first panoramic image. A mesh representation of the first panoramic image may be used instead of a point cloud representation, because this can avoid density problems associated with generating a point cloud from an ERP image. For example, a point cloud generated from an ERP image may contain large changes in sparseness when traveling large distances, which can make inpainting (e.g., filling in holes in arbitrary topology so that the additions appear to be part of the original image) difficult.

[0069] In step S920, a composite panoramic image is generated by fusing the first difference prediction with the second difference prediction. For example, a fusing network (e.g., fusing network 265) can fuse an RGB-D image (e.g., RGB255-1) associated with the first difference mesh with an RGB-D image (RGB260-1) associated with the second difference mesh. One or more RGB-Ds may contain holes due to occlusion in the composite view at target positions 245, 250. Thus, the fusing may include inpainting of holes. The fusing can generate a composite panorama using a trained model (e.g., a trained neural network). The trained neural network may include seven downsampling elements and seven upsampling elements. In one exemplary implementation, the fusing may include a step in each of the RGB-Ds to identify holes (e.g., negative regions in the mesh rendering depth image) by generating a binary visibility mask based on visibility maps (e.g., visibility 255-2, 260-2). The fusion process may involve a step that simulates a circular convolutional neural network (CNN) by using circular padding in each convolutional layer to join the left and right ends. Zero padding can be used at the beginning and end of each feature map.

[0070] In step S925, the loss is calculated based on the composite panoramic image and the third panoramic image in the sequence of panoramic images. For example, the third panoramic image (e.g., panorama 430-2) may be sequentially between the first panoramic image (e.g., panorama 430-1) and the second panoramic image (e.g., panorama 430-3). The loss can be calculated with respect to loss 440 as described above.

[0071] Training a fusion network may involve a step using a sequence of three panoramas (e.g., panoramas 430-1, 430-2, and 430-3). Using the pose of the intermediate panorama (panorama 430-2), a mesh rendering may be generated from the first panorama (panorama 430-1) and the last panorama (panorama 430-3). The fusion network can receive the mesh rendering and combine it to predict the intermediate panorama (e.g., panorama 270). The intermediate panorama (e.g., panorama 430-2), which is the ground truth, may be used for supervision. The loss may be used to train the fusion network.

[0072] In step S930, the panoramic image fusion is trained based on the loss. For example, training the fusion network may include a step of changing the weights associated with at least one convolution related to the fusion network. In one exemplary implementation, the fusion network may be trained based on the difference between the predicted panorama (e.g., panorama 270) and the ground truth panorama (e.g., panorama 430-2). The loss (e.g., loss 440) may be generated based on the difference between the predicted panorama and the ground truth panorama. Training may be repeated until the loss is minimized and / or until there are no significant changes in the loss between iterations. In one exemplary implementation, a smaller loss results in better synthesis (e.g., prediction) of the intermediate panoramas by the fusion network.

[0073] Figure 10 shows a block diagram of a computing system according to at least one exemplary embodiment. As shown in Figure 10, the computing system includes at least one processor 1005 and at least one memory 1010. The at least one memory 1010 may include at least depth prediction blocks 225, a differential mesh renderer 235, and a fusion network.

[0074] In the example of Figure 10, the computing system should be understood to substantially represent any computing device configured to perform the techniques described herein, which may consist of at least one computing device or include at least one computing device. Therefore, the computing system should be understood to include various components that may be used to implement the techniques described herein or different or future versions thereof. For example, the computing system is illustrated to include at least one processor 1005 and at least one memory 1010 (e.g., a non-temporary computer-readable storage medium).

[0075] At least one processor 1005 may be used to execute instructions stored in at least one memory 1010. Thus, at least one processor 1005 may implement various features and functions described herein, or additional or alternative features and functions. At least one processor 1005 and at least one memory 1010 may be used for various other purposes. For example, at least one memory 1010 may represent an example of various types of memory and associated hardware and software that may be used to implement any of the modules described herein.

[0076] At least one memory 1010 may be configured to store data and / or information related to the computing system. At least one memory 1010 may be a shared resource. For example, the computing system may be an element of a larger system (e.g., a server, personal computer, mobile device, and / or similar). Thus, at least one memory 1010 may be configured to store data and / or information related to other elements within the larger system (e.g., image / video handling, web browsing, or wired / wireless communication).

[0077] Figure 11 shows examples of computer devices 1100 and mobile computer devices 1150 that may be used with the technologies described herein. Computing device 1100 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 1150 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are illustrative only and are not intended to limit the forms of implementation of the invention described herein and / or claimed herein.

[0078] The computing device 1100 includes a processor 1102, memory 1104, storage device 1106, a high-speed interface 1108 connected to memory 1104 and a high-speed expansion port 1110, and a low-speed bus 1114 and a low-speed interface 1112 connected to storage device 1106. Each of the components 1102, 1104, 1106, 1108, 1110, and 1112 may be interconnected using various buses and implemented on a common motherboard, or in other ways as needed. The processor 1102 can process instructions, including instructions stored in memory 1104 or on storage device 1106, to run within the computing device 1100, and display graphical information for a GUI on an external input / output device such as a display 1116 coupled to the high-speed interface 1108. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and memory types, as needed. Furthermore, multiple computing devices 1100 may be connected, each device contributing to the required operation (for example, as a server bank, a group of blade servers, or a multiprocessor system).

[0079] Memory 1104 stores information within the computing device 1100. In one implementation, memory 1104 is one or more volatile memory units. In another implementation, memory 1104 is one or more non-volatile memory units. Memory 1104 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0080] The storage device 1106 can provide a large-capacity storage device for the computing device 1100. In one implementation, the storage device 1106 may be an array of devices including computer-readable media such as floppy disk drives, hard disk drives, optical disk drives, or tape drives, flash memory or other similar semiconductor memory devices, or devices in a storage area network or other configuration. Computer program products may be materially embodied in an information medium. Computer program products may also include instructions that, when executed, perform one or more methods, such as those described above. The information medium is a computer-readable or machine-readable medium such as memory 1104, the storage device 1106, or memory on the processor 1102.

[0081] The high-speed controller 1108 manages the high-bandwidth operations of the computing device 1100, while the low-speed controller 1112 manages operations that do not require such high bandwidth. Such a functional allocation is illustrative only. In one implementation, the high-speed controller 1108 is coupled to memory 1104, a display 1116 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 1110 that accepts various expansion cards (not shown). In this implementation, the low-speed controller 1112 is coupled to the storage device 1106 and the low-speed expansion port 1114. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth®, Ethernet®, wireless Ethernet) and may be coupled to one or more input / output devices such as a keyboard, pointing device, scanner, or to a network device such as a switch or router via a network adapter, for example.

[0082] The computing device 1100 can be implemented in several separate forms, as shown in the figure. For example, the computing device 1100 can be implemented as a standard server 1120, or often as a group of such servers. Alternatively, the computing device 1100 can be implemented as part of a rack server system 1124. In addition, the computing device 1100 can be implemented as a personal computer, such as a laptop computer 1122. Alternatively, the components of the computing device 1100 can be combined with other components of a mobile device (not shown), such as device 1150. Each of such devices may contain one or more of the computing devices 1100, 1150, and the overall system may consist of multiple computing devices 1100, 1150 communicating with each other.

[0083] The computing device 1150 includes, among other components, a processor 1152, memory 1164, input / output devices such as a display 1154, a communication interface 1166, and a transceiver 1168. The device 1150 may also include storage devices such as a microdrive or other devices to provide additional storage. Each of the components 1150, 1152, 1164, 1154, 1166, and 1168 is interconnected using various buses, and some of the components may be implemented on a common motherboard, or in other ways as needed.

[0084] The processor 1152 can execute instructions, including those stored in memory 1164, within the computing device 1150. The processor may be implemented as a chipset of chips, including multiple individual analog and digital processors. The processor may coordinate other components of device 1150, such as controlling the user interface, controlling applications run by device 1150, or controlling wireless communication by device 1150.

[0085] The processor 1152 can communicate with the user by controlling interface 1158 and display interface 1156, which are coupled to the display 1154. The display 1154 may be, for example, a thin-film transistor liquid crystal display (TFT LCD) or an organic light-emitting diode (OLED) display, or other suitable display technology. The display interface 1156 may include appropriate circuitry for driving the display 1154 to present graphic information or other information to the user. The control interface 1158 may receive commands from the user and translate those commands for presentation to the processor 1152. In addition, an external interface 1162 that communicates with the processor 1152 may be provided to enable proximity communication of device 1150 with other devices. The external interface 1162 may, for example, provide wired communication in some implementations and wireless communication in other implementations, and multiple interfaces may be used.

[0086] Memory 1164 stores information within the computing device 1150. Memory 1164 may be implemented as one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. Additional memory 1174 may also be provided and may be connected to device 1150 via an expansion interface 1172, which may include, for example, a single in-line memory module (SIMM) card interface. Such additional memory 1174 may provide device 1150 with additional storage space or may also store applications or other information for device 1150. Specifically, additional memory 1174 may include instructions for performing or supplementing the operations described above, and may also include security information. Thus, additional memory 1174 may be provided, for example, as a security module for device 1150 and may be programmed with instructions that enable the secure use of device 1150. In addition, a SIMM card may provide a secure application with additional information, such as setting identification information on the SIMM card in a hack-proof manner.

[0087] The memory may include, for example, flash memory and / or NVRAM memory, as discussed below. In one implementation, the computer program product is materially embodied in an information medium. The computer program product may contain instructions that, when executed, perform one or more methods, such as those described above. The information medium is a computer-readable or machine-readable medium, such as memory 1164, expansion memory 1174, or memory on processor 1152, which may be housed in transceiver 1168 or external interface 1162.

[0088] Device 1150 can communicate wirelessly via a communication interface 1166, which may include digital signal processing circuitry if necessary. The communication interface 1166 can facilitate communication under various modes or protocols, including, among others, GSM® voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA®, CDMA2000, or GPRS. Such communication may occur, for example, via a radio frequency transceiver 1168. In addition, short-range communication may occur using Bluetooth, Wi-Fi, or other such transceivers (not shown). Furthermore, a Global Positioning System (GPS) receiver module 1170 may supply device 1150 with additional radio data related to navigation and location, which may be used as needed by applications operating on device 1150.

[0089] Device 1150 may communicate using a speech-recognizable voice codec 1160 that can receive spoken information from the user and convert that information into usable digital information. The voice codec 1160 can also generate audible sound for the user, for example, through a speaker in the handset of device 1150. Such sound may include sound from voice calls, recorded sound (e.g., voicemail, music files), and sound generated by applications running on device 1150.

[0090] The computing device 1150 can be implemented in several separate forms, as shown in the figure. For example, the computing device 1150 can be implemented as a mobile phone 1180. The computing device 1150 may also be implemented as part of a smartphone 1182, a personal digital information processing terminal, or other similar mobile device.

[0091] The exemplary embodiments may include various modifications and alternative forms, which are shown in the figures as examples and described in detail herein. However, it should be understood that the exemplary embodiments are not intended to limit themselves to any particular form disclosed, but rather to include all modifications, equivalents, and alternative forms included in the claims. Throughout the description of the figures, similar numbers refer to similar elements.

[0092] The various implementations of the systems and technologies described herein can be realized in digital electronic circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations of one or more computer programs that are executable and / or translatable into machine language in a programmable system including at least one programmable processor, the programmable processor may be dedicated or general-purpose, and may be coupled with a storage system, at least one input device, and at least one output device to exchange data and instructions. The various implementations of the systems and technologies described herein may be realized and / or generally referred to herein as circuits, modules, blocks, or systems that can combine software and hardware aspects. For example, a module may include functions / actions / computer program instructions to be executed in a processor (e.g., a processor formed on a silicon substrate, a GaAs substrate, etc.) or several other programmable data processing devices.

[0093] Some of the embodiments described above are explained as processes or methods represented as flowcharts. In flowcharts, operations are explained as sequential processes, but many operations can be performed in parallel, together, or simultaneously. In addition, the order of operations may be rearranged. A process may terminate when its operation is complete, but it may have additional steps not shown in the diagram. A process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0094] The methods discussed above, some of which are illustrated in flowcharts, can be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. Program code or code segments that perform the required tasks, when implemented in software, firmware, middleware, or microcode, can be stored in machine-readable or computer-readable media such as storage media. One or more processors can perform the required tasks.

[0095] The specific structural and functional details disclosed herein are merely illustrative to illustrate the exemplary embodiments. However, the exemplary embodiments are embodied in many alternative forms and should not be construed as being limited solely to the embodiments described herein.

[0096] In this specification, terms such as “first,” “second,” etc., may be used to describe various elements, but it should be understood that these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, without departing from the scope of the exemplary embodiments, the first element may be referred to as the second element, and similarly, the second element may be referred to as the first element. The term “and / or” as used herein includes any combination of one or more of the items listed relatedly, or all of them.

[0097] When an element is said to be connected to or joined to another element, it should be understood that it may be directly connected to or joined to the other element, or there may be an intervening element. In contrast, when an element is said to be directly connected to or joined to another element, there is no intervening element. Other words used to describe the relationship between elements (e.g., "between" vs. "directly between," "proximity" vs. "adjacent") should be interpreted similarly.

[0098] The terminology used herein is intended solely to describe specific embodiments and is not intended to limit the exemplary embodiments. The singular forms used herein, such as “a,” “an,” and “the,” are intended to include the plural form unless the context explicitly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” as used herein, specify the presence of expressly stated features, completes, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, completes, steps, actions, elements, components, and / or groups thereof.

[0099] It should also be noted that in some alternative implementations, the functions / actions shown may occur in a different order than those shown in the diagrams. For example, two diagrams shown consecutively may actually be executed simultaneously, depending on the functions / actions they encompass, and sometimes even in reverse order.

[0100] All terms used herein (including technical and scientific terms) have the same meaning as generally understood by those skilled in the art to which the exemplary embodiments belong, unless otherwise defined. For example, terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the context of the art in question, and therefore, unless explicitly defined herein, should not be interpreted in an idealized or overly formalized sense.

[0101] Some of the exemplary embodiments described above and the corresponding detailed descriptions relate to algorithms and symbolic representations of operations on data bits within software or computer memory. These descriptions and representations are intended to effectively convey the substance of one's work to others skilled in the art. The term algorithm as commonly used herein is understood to be a self-consistent sequence of steps that lead to a desired result. The steps require the physical manipulation of physical quantities. While not mandatory, these quantities typically take the form of optical, electrical, or magnetic signals that can be stored, transferred, combined, compared, or manipulated. These signals have proven convenient, primarily for reasons of common usage, sometimes being referred to as bits, values, elements, symbols, characteristics, terms, numbers, etc.

[0102] In the exemplary embodiments described above, references to actions and symbolic representations of behavior (e.g., in the form of flowcharts), which may be implemented as program modules or functional processes, include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types, and may be described and / or implemented using existing hardware in existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate array (FPGA) computers, etc.

[0103] However, it should be kept in mind that all these terms and similar terms should be associated with appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise specified, and as is evident from the discussion, terms such as processing, operation, calculation, determination, and display refer to the actions and operations of a computer system or similar computing device that process or convert data, represented as physical and electronic quantities within the registers and memory of a computer system, into other data, similarly represented as physical quantities, within the memory or registers of a computer system, or other storage, transmission, or display devices of such information.

[0104] It should also be noted that the embodiments implemented in the software of the exemplary embodiments are generally encoded on some form of non-temporary program storage medium or implemented through some type of transmission medium. The program storage medium may be a magnetic medium (e.g., a floppy disk or hard drive) or an optical medium (e.g., a compact disk read-only memory i.e., CD-ROM), and may be a read-only medium or a random-access medium. Similarly, the transmission medium may be twisted-pair wire, coaxial cable, optical fiber, or other suitable transmission medium known in the art. The exemplary embodiments are not limited by these embodiments of any given implementation form.

[0105] Finally, while the attached claims present specific combinations of the features described herein, it should be noted that the scope of this disclosure is not limited to the specific combinations claimed below, but extends to encompass all combinations of the features or embodiments disclosed herein, regardless of whether such specific combinations are specifically enumerated in the attached claims at this time.

Claims

1. The process includes the step of predicting the stereo depth associated with a first panoramic image and a second panoramic image, A step of generating a first mesh representation based on the first panoramic image and the stereo depth corresponding to the first panoramic image, A step of generating a second mesh representation based on the second panoramic image and the stereo depth corresponding to the second panoramic image, The steps include: applying the first mesh representation and the second mesh representation to a trained model to generate a third panoramic image to be inserted between the first panoramic image and the second panoramic image; The trained model includes, The model predicts the intermediate panoramic image in a sequence of three panoramic images, which includes a first image, a second image, and an intermediate panoramic image existing between the first and second images. In order to reduce the difference between the ground truth image and the image predicted as the intermediate panoramic image by the model, the settings in the model are changed, A method generated by

2. The method according to claim 1, wherein the first panoramic image and the second panoramic image are 360-degree, wide baseline equirectangular (ERP) panoramas.

3. The method according to claim 1, wherein the step of predicting the stereo depth is to estimate the depth of the first panoramic image and the second panoramic image using a spherical sweep cost volume configured to align the feature map associated with the first panoramic image with the feature map associated with the second panoramic image.

4. The step of predicting the stereo depth involves estimating the low-resolution depth based on a first feature map associated with the first panoramic image and the second panoramic image. The step of predicting the stereo depth involves estimating high-resolution depth based on the first feature map and a second feature map associated with the first panoramic image. The method according to claim 1.

5. The step of generating the first mesh representation is based on the first panoramic image and discontinuities determined based on the stereo depth corresponding to the first panoramic image, The step of generating the second mesh representation is based on the second panoramic image and discontinuities determined based on the stereo depth corresponding to the second panoramic image, The method according to claim 1.

6. The first panoramic image is rendered from a viewpoint corresponding to the first target position, The step of generating the first mesh representation includes rendering the first mesh representation into a first 360-degree panorama based on the first target position. The aforementioned second panoramic image is rendered from a viewpoint corresponding to the second target position. The step of generating the second mesh representation includes rendering the second mesh representation to a first 360-degree panorama based on the second target position. The method according to claim 1.

7. The step of generating the third panoramic image includes identifying at least one hole associated with the generation of the third panoramic image, wherein the hole is represented as a negative value in the depth map, and the step of generating the third panoramic image further includes, The method according to claim 1, further comprising the step of inpainting the at least one hole in the generated third panoramic image.

8. The system includes a depth predictor configured to predict the stereo depth associated with a first panoramic image and a second panoramic image, A first differential mesh renderer configured to generate a first mesh representation based on the first panoramic image and the stereo depth corresponding to the first panoramic image, A second differential mesh renderer is configured to generate a second mesh representation based on the second panoramic image and the stereo depth corresponding to the second panoramic image, A fusion network configured to generate a third panoramic image to be inserted between the first and second panoramic images by applying the first and second mesh representations to a trained model, The trained model is equipped with, The model predicts the intermediate panoramic image in a sequence of three panoramic images, which includes a first image, a second image, and an intermediate panoramic image existing between the first and second images. In order to reduce the difference between the ground truth image and the image predicted as the intermediate panoramic image by the model, the settings in the model are changed, A system generated by [the system].

9. The system according to claim 8, wherein the first panoramic image and the second panoramic image are 360-degree, wide baseline equirectangular projection (ERP) panoramas.

10. The system according to claim 8, wherein predicting the stereo depth involves estimating the depth of the first panoramic image and the second panoramic image, respectively, using a spherical sweep cost volume configured to align the feature map associated with the first panoramic image with the feature map associated with the second panoramic image.

11. Predicting the stereo depth involves estimating the low-resolution depth based on a first feature map associated with the first panoramic image and the second panoramic image. Predicting the stereo depth involves estimating high-resolution depth based on the first feature map and a second feature map associated with the first panoramic image. The system according to claim 8.

12. The generation of the first mesh representation is based on the first panoramic image and the discontinuities determined based on the stereo depth corresponding to the first panoramic image. The generation of the second mesh representation is based on the second panoramic image and discontinuities determined based on the stereo depth corresponding to the second panoramic image. The system according to claim 8.

13. The first panoramic image is rendered from a viewpoint corresponding to the first target position, Generating the first mesh representation includes rendering the first mesh representation into a first 360-degree panorama based on the first target position. The aforementioned second panoramic image is rendered from a viewpoint corresponding to the second target position. The generation of the second mesh representation includes rendering the second mesh representation to a first 360-degree panorama based on the second target position. The system according to claim 8.

14. The trained model uses circular padding in each convolutional layer to join the left and right edges of the third panoramic image. The system according to claim 8.

15. A computer program including instructions, wherein when the instructions are executed by at least one processor of a computer system, the computer system The system is configured to predict the stereo depth associated with a first panoramic image and a second panoramic image, wherein the first and second panoramic images are 360-degree, wide baseline equirectangular projection (ERP) panoramas. A first mesh representation is generated based on the first panoramic image and the stereo depth corresponding to the first panoramic image. A second mesh representation is generated based on the second panoramic image and the stereo depth corresponding to the second panoramic image. By applying the first mesh representation and the second mesh representation to a trained model, a third panoramic image is generated that is inserted between the first panoramic image and the second panoramic image. The trained model is configured to perform the following actions: The model predicts the intermediate panoramic image in a sequence of three panoramic images, which includes a first image, a second image, and an intermediate panoramic image existing between the first and second images. In order to reduce the difference between the ground truth image and the image predicted as the intermediate panoramic image by the model, the settings in the model are changed, A computer program generated by [a specific entity / organization].

16. The first panoramic image is rendered from a viewpoint corresponding to the first target position, Generating the first mesh representation includes rendering the first mesh representation into a first 360-degree panorama based on the first target position. The aforementioned second panoramic image is rendered from a viewpoint corresponding to the second target position. The generation of the second mesh representation includes rendering the second mesh representation to a first 360-degree panorama based on the second target position. The positions of each gap in the second panoramic image are as follows: The computer program according to claim 15.

Images