Generating panoramic videos

Abstract

The media application receives an input panning video including image frames captured by a camera that pans at least once from a first side to a second side. The media application further includes reprojecting the input panning video onto a panoramic canvas, the panoramic canvas including the image frames and the missing region. The media application further includes generating a base video including synthesized pixels of the missing region at a coarse time scale, and outputting the panoramic video including synthesized pixels of the missing region of the panoramic canvas using a generative machine learning model by performing temporal upsampling, merging the temporally upsampled base video with the panoramic canvas, and refining the base video by resynthesizing the synthesized pixels of the missing region in the base video.

Description

[Technical Field]

[0001] Cross-reference of related applications This application is an international application claiming priority under 119(e) of U.S. Patent Act to U.S. Provisional Patent Application No. 63 / 616,037, “Generating Panoramic Videos,” filed on 29 December 2023, which is incorporated herein by reference in its entirety. [Background technology]

[0002] Panoramic stitching refers to simulating a wider field of view image from a set of images captured by rotating a camera in place. For example, a set of images can be joined when the camera moves 180 degrees along the same axis. Panoramic stitching includes sparse or semi-dense alignment, rigid alignment or depth-compensated alignment, and image fusion to resolve seams and discrepancies across the entire observation.

[0003] Because scene movement can cause failures in both alignment and compositing, panoramic videos are more complex than panoramic images, especially when they contain dynamic objects. Furthermore, when the camera captures an image from a first position along the axis, other images are not captured from other positions along the axis, resulting in missing parts of the panoramic video, making it impossible to create panoramic videos from images using conventional systems.

[0004] The background art provided herein is intended to provide a general overview of the context of this disclosure. Within the scope of the background art section, the work of the inventors named herein, as well as aspects of this specification that may not meet the requirements of prior art at the time of filing, are not expressly or implicitly recognized as prior art to this disclosure. [Overview of the Initiative]

[0005] A method performed by a computer includes receiving an input panning video containing image frames captured by a camera panning at least once from a first side to a second side. The method further includes reprojecting the input panning video onto a panoramic canvas, the panoramic canvas containing image frames and missing regions. The method further includes generating a base video containing the synthesized pixels of the missing regions on a coarse time scale, and refining the base video by performing time upsampling, merging the time-upsampled base video with the panoramic canvas, and recombining the synthesized pixels of the missing regions in the base video, thereby outputting a panoramic video containing the synthesized pixels of the missing regions on the panoramic canvas using a generative machine learning model. In some embodiments, the method performed by a computer is performed as a media application on a computing device.

[0006] In some embodiments, time upsampling includes temporally upsampling a base video and performing spatial and color alignment of the temporally upsampled base video with a panoramic canvas to obtain an aligned temporally upsampled base video. In some embodiments, merging the temporally upsampled base video with a panoramic canvas includes merging the aligned temporally upsampled base video with the panoramic canvas. In some embodiments, recomposition includes generating a mask of pixels in the missing areas, regenerating composite pixels in the missing areas based on the mask, and merging the regenerated pixels with the composite pixels. In some embodiments, the quality of the panoramic video improves as a function of the number of times the image frames captured by the camera pan from a first side to a second side and from a second side to a first side. In some embodiments, the input panning video includes one or more moving objects. In some embodiments, the method further includes applying a super-resolution pass to the panoramic video in response to outputting the panoramic video to increase the resolution of the panoramic video. In some embodiments, the method further includes selecting a panoramic image from the panoramic video.

[0007] In some embodiments, a non-temporary computer-readable medium containing instructions causing one or more computers to perform an operation, when executed by one or more computers. The operation includes receiving an input panning video, which includes camera-captured image frames panning at least once from a first side to a second side, and reprojecting the input panning video onto a panoramic canvas, which includes image frames and missing regions. The operation further includes generating a base video, which includes pixels of the missing regions, on a coarse time scale, and outputting a panoramic video, which includes composite pixels of the missing regions of the panoramic canvas, by performing time upsampling, merging the time-upsampled base video with the panoramic canvas, and refining the base video by recombining the composite pixels of the missing regions in the base video, using a generative machine learning model.

[0008] In some embodiments, time upsampling includes temporally upsampling a base video and performing spatial and color alignment of the temporally upsampled base video with a panoramic canvas to obtain an aligned temporally upsampled base video. In some embodiments, merging the temporally upsampled base video with a panoramic canvas includes merging the aligned temporally upsampled base video with the panoramic canvas. In some embodiments, recomposition includes generating a mask of pixels in the missing areas, regenerating composite pixels in the missing areas based on the mask, and merging the regenerated pixels with the composite pixels. In some embodiments, the input panning video includes one or more moving objects. In some embodiments, the operation further includes applying a super-resolution pass to the panoramic video in response to outputting a panoramic video to increase the resolution of the panoramic video. In some embodiments, the operation further includes selecting a panoramic image from the panoramic video.

[0009] In some embodiments, the system includes one or more processors and a memory coupled to one or more processors, the memory storing instructions that, when executed by the processors, cause the processors to perform an operation. The operation includes receiving an input panning video, which includes a camera-captured image frame panning at least once from a first side to a second side, and reprojecting the input panning video onto a panoramic canvas, which includes an image frame and missing regions. The operation further includes generating a base video, which includes pixels of the missing regions, on a coarse time scale, and outputting a panoramic video, which includes synthesized pixels of the missing regions of the panoramic canvas, by performing temporal upsampling, merging the temporally upsampled base video with the panoramic canvas, and refining the base video by recombining the synthesized pixels of the missing regions in the base video, using a generative machine learning model.

[0010] In some embodiments, time upsampling includes temporally upsampling a base video and performing spatial and color alignment of the temporally upsampled base video with a panoramic canvas to obtain an aligned temporally upsampled base video. In some embodiments, merging the temporally upsampled base video with a panoramic canvas includes merging the aligned temporally upsampled base video with the panoramic canvas. In some embodiments, recomposition includes generating a mask of pixels in the missing areas, regenerating composite pixels in the missing areas based on the mask, and merging the regenerated pixels with the composite pixels. In some embodiments, the input panning video includes one or more moving objects. In some embodiments, the operation further includes applying a super-resolution pass to the panoramic video in response to outputting a panoramic video to increase the resolution of the panoramic video. In some embodiments, the operation further includes selecting a panoramic image from the panoramic video. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram of an exemplary network environment according to some embodiments described herein. [Figure 2] This is a block diagram of an exemplary computing device according to some embodiments described herein. [Figure 3] The following are examples of several embodiments described herein, showing the input image from an input panning video, the input image projected onto a panoramic canvas, and the output image from the resulting output panoramic video. [Figure 4] This specification describes several embodiments of an exemplary token-based model that outputs panoramic video. [Figure 5]This specification describes exemplary diffusion models trained to output panoramic video, according to several embodiments described herein. [Figure 6A] This specification describes exemplary processes for generating panoramic videos according to several embodiments described herein. [Figure 6B] This specification describes exemplary processes for generating panoramic videos according to several embodiments described herein. [Figure 6C] This specification describes exemplary processes for generating panoramic videos according to several embodiments described herein. [Figure 6D] This specification describes exemplary processes for generating panoramic videos according to several embodiments described herein. [Figure 7] This specification illustrates an example of how a diffusion model performs upsampling and outpainting to generate panoramic video, based on several embodiments described herein. [Figure 8] This specification provides an illustrative comparison of how token-based models are used compared to diffusion models in terms of probabilities during outpainting, based on several embodiments described herein. [Figure 9] A flowchart shows an exemplary method for generating panoramic video according to some embodiments described herein. [Modes for carrying out the invention]

[0012] overview When a user captures an input video by panning a camera left and right (or in any direction to capture different regions), the resulting panoramic video may have more synthesized pixels than the original pixels. This further increases the need for image generation to appropriately track the movement of objects within the panoramic video. For example, a panoramic video of people kayaking in a river requires not only image generation of stationary objects such as buildings along the horizon, but also tracking and alignment of movement, such as the movement of the paddles in the water. Also, the task is more difficult when the camera capturing the input video is moving during capture.

[0013] The techniques described herein advantageously generate a panoramic video by reprojection of an input panning video onto a panoramic canvas, the panoramic canvas including image frames from the input panning video and missing regions. A generative machine learning model outputs a panoramic video including synthesized pixels for the missing regions of the panoramic canvas. For example, the generative machine learning model may generate a base video including synthesized pixels for the missing regions on a coarse time scale. In some embodiments, the generation model (and / or other techniques) may refine the base video by performing temporal upsampling, merging the temporally upsampled base video with input pixels from the panoramic canvas, and resynthesizing the synthesized pixels for the missing regions. The techniques may be used with videos including moving objects, moving people, portrait panoramic pans, etc.

[0014] Exemplary computing device FIG. 2 is a block diagram of an exemplary computing device 200 that may be used to implement one or more features described herein. Computing device 200 may be any suitable computer system, server, or other electronic or hardware device. In one example, computing device 200 is media server 101. In other examples, computing device 200 is user device 115.

[0015] In some embodiments, computing device 200 includes a processor 235, a memory 237, an input / output (I / O) interface 239, a display 241, a camera 243, and a storage device 245, all coupled via a bus 218. Processor 235 may be coupled to bus 218 via signal line 222, memory 237 may be coupled to bus 218 via signal line 224, I / O interface 239 may be coupled to bus 218 via signal line 226, display 241 may be coupled to bus 218 via signal line 228, camera 243 may be coupled to bus 218 via signal line 230, and storage device 245 may be coupled to bus 218 via signal line 232.

[0016] The processor 235 may be one or more processors and / or processing circuits that execute program code and control the basic operation of the computing device 200. "Processor" includes any suitable hardware system, mechanism, or component that processes data, signals, or other information. A processor may include a general-purpose central processing unit (CPU) having one or more cores (e.g., single-core, dual-core, or multi-core configuration), multiple processing units (e.g., multi-processor configuration), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), composite programmable logic devices (CPLDs), dedicated circuits for realizing a function, dedicated processors for performing processing based on neural network models, neural circuits, processors optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some embodiments, the processor 235 may include one or more coprocessors that perform neural network processing. In some embodiments, the processor 235 may be a processor that processes data to produce a probabilistic output, for example, the output produced by the processor 235 may be inaccurate or accurate within a range from an expected output. The processing does not need to be limited to a specific geographical location or have temporal constraints. For example, the processor may perform its functions in real time, offline, or batch mode. Parts of the processing may be performed at different times and locations by different (or the same) processing systems. The computer can be any processor that communicates with memory.

[0017] Memory 237 may be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erase-read-only memory (EEPROM), or flash memory, which is typically located separately from and / or integrated with the processor 235, and is provided in the computing device 200 for access by the processor 235 and is suitable for storing instructions for execution by the processor or a set of processors. Memory 237 can store software running on the computing device 200 by the processor 235, including media applications 103.

[0018] Memory 237 may include an operating system 262, other applications 264, and application data 266. Other applications 264 may include, for example, an image library application, an image management application, an image gallery application, a communication application, a web hosting engine or application, a media sharing application, and the like. One or more methods disclosed herein can operate in several environments and platforms, for example, as a standalone computer program that can run on any type of computing device, as a web application having web pages, as a mobile application ("App") that runs on a mobile computing device, and the like.

[0019] Application data 266 may also be data generated by other applications 264 or the hardware of the computing device 200. For example, application data 266 may include images used by an image library application and user actions identified by other applications 264 (e.g., a social networking application).

[0020] The I / O interface 239 can provide functionality that enables the computing device 200 to interface with other systems and devices. Interfaced devices may be included as part of the computing device 200, or they may be separate and communicate with the computing device 200. For example, network communication devices, storage devices (e.g., memory 237 and / or storage device 245), and input / output devices can communicate via the I / O interface 239. In some embodiments, the I / O interface 239 can be connected to interface devices such as input devices (keyboards, pointing devices, touchscreens, microphones, scanners, sensors, etc.) and / or output devices (display devices, speaker devices, printers, monitors, etc.).

[0021] Some examples of interface devices that can be connected to the I / O interface 239 include a display 241 that can be used to display content, such as images, videos, and / or the user interface of an output application as described herein, and to receive touch (or gesture) input from the user. For example, the display 241 may be used to display a user interface. The display 241 may include any suitable display device, such as a liquid crystal display (LCD), light-emitting diode (LED), or plasma display screen, cathode ray tube (CRT), television, monitor, touchscreen, three-dimensional display screen, or other visual display device. For example, the display 241 may be a flat display screen provided on a mobile device, a shape factor of eyeglasses or multiple display screens embedded in a headset device, or a monitor screen of a computer device.

[0022] Camera 243 can be any type of image capture device capable of capturing images and / or video. In some embodiments, camera 243 captures images or video that the I / O interface 239 sends to the media application 103.

[0023] The storage device 245 stores data related to the media application 103. For example, the storage device 245 may store a training dataset containing training data such as multiple training images, a generative machine learning model, and an outpainter model.

[0024] The media application 103 receives an input panning video containing image frames captured by camera 243 panning at least once from a first side to a second side. In some embodiments, camera 243 captures multiple pans in a single capture, such as panning from a first side to a second side and then back to the first side. For example, a user may acquire a user device 115 and move it left and right multiple times to capture the input video. The resolution of the panoramic video is improved by multiple pans to enhance motion tracking at different spatial locations.

[0025] As the number of image frames increases, the number of composite pixels generated for missing areas in the panoramic video decreases, thus improving the resolution of the panoramic video by the number of image frames captured at different angles. In some embodiments, the input panning video includes one or more moving objects such as people in motion (e.g., people kayaking in a river, people scuba diving, people walking around a landmark, people skating), cars, trees, water, etc.

[0026] The media application 103 reprojects the input panning video onto the panoramic canvas and converts the input frame (x) into the panoramic coordinate system. 0 ) and the corresponding mask frame (m 0) generates. Masks can be used to identify valid pixels to be retained and missing areas where composite pixels will be generated. A panoramic canvas is a canvas with panoramic dimensions that includes image frames and missing areas (scenes that the camera did not capture within the image frames captured at that moment). The panoramic canvas includes both image frames from the input panning video positioned based on coordinate positions within the panoramic canvas, and missing areas that may be represented as blank pixels, black pixels, or no data at all.

[0027] Figure 3 shows three input images 300 from an input panning video, three input images 300 projected onto a panoramic canvas 310, and three output images 320 from an output panoramic video. The input images 300 show people kayaking on a river. The first input image 300a includes a person 301 in a kayak in the foreground, a person 302 in a kayak in the background, and the horizon 303. The second input image 300b includes the first person 301 at a later point in time in the input panning video, with the paddle 304 in a different position compared to the first input image 300a, and the kayak 305 is also visible. The third input image 300c includes the second person 302 in a different position compared to the first input image 300a.

[0028] The panoramic canvas 310 includes each input image 300 reprojected onto the panoramic canvas 310. In some embodiments, the media application 103 uses a projection transform solver to perform the reprojection of the input panning video, which includes rotation but does not include translation or scaling. In response to determining that parallax correction is required, the media application 103 may use simultaneous localization mapping (SLAM) to perform the reprojection. In some embodiments, the media application 103 ignores the translation of the camera 243, calculates the altitude and azimuth of each input pixel ray with respect to the direction of the corresponding camera 243, and then projects each ray onto the equirectangular canvas.

[0029] The output image 320 includes composite pixels from the missing areas of the panoramic canvas 310. For example, the second output image 320b includes a second person 302 that was not visible in the second input image 300b.

[0030]

number

[0031] In some embodiments, the input panning video is wider than the native aspect ratio of the generative machine learning model. The media application 103 adjusts the input panning video (x) to match the native height of the generative machine learning model. K ) can be downscaled.

[0032] Generative machine learning models The generative machine learning model generates a base video containing synthesized pixels from missing regions on a coarse timescale. In some embodiments, the generative machine learning model outputs multiple overlapping spatial windows across the panoramic width. This process is described in more detail below with reference to Figures 6A to 6D, Figure 7, and Figure 8. The generative machine learning model may average the distribution predicted by the generative machine learning model for each window and output a base image frame of the base video based on the average.

[0033] The generative machine learning model refines the base video using time upsampling, merging the base video with input pixels from the input panning video, and reconstructing the synthesized pixels in the missing regions.

[0034] The media application 103 uses training data to train a generative machine learning model to output a panoramic video from input data, such as application data 266 (e.g., input panning video captured by the user device 115). The training data can be obtained from any source, such as a data repository explicitly designated for training, or data for which permission has been granted to be used as training data for machine learning. In some embodiments, training may take place on a media server 101 that directly provides the training data to the user device 115, or training may take place locally on the user device 115, or a combination of both.

[0035] A trained machine learning model may include one or more model forms or structures. For example, a model form or structure may include any type of neural network, such as a linear network, a deep learning neural network implementing multiple layers (e.g., a "hidden layer" between an input layer and an output layer, each of which is a linear network), a convolutional neural network (e.g., a network that divides or partitions input data into multiple parts or tiles, processes each tile individually using one or more neural network layers, and aggregates the results of processing each tile), or a sequence-to-sequence neural network (e.g., a network that takes sequential data such as words in a sentence or frames in a video as input and produces a resulting sequence as output).

[0036] The model format or structure may specify the connectivity between various nodes and the organization of the nodes into layers. For example, the nodes of the first layer (e.g., the input layer) may receive data as input data or application data. Such data may include, for example, one or more pixels per node when the trained model is used, for example, to analyze an image. Subsequent intermediate layers may receive the output of the nodes of the previous layer as input, according to the connectivity specified in the model format or structure. These layers may also be called hidden layers. The final layer (e.g., the output layer) produces the output of the generative machine learning model. In some embodiments, the model format or structure also specifies the number and / or type of nodes in each layer.

[0037] In different embodiments, a trained generative machine learning model may include one or more models. One or more of the models may include multiple nodes arranged in layers according to the model structure or form. In some embodiments, a node may be a memoryless computation node configured to process one unit of input and produce one unit of output. The computations performed by the node may include, for example, multiplying each of the multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum with a bias or intercept value to produce a node output. In some embodiments, the computations performed by the node may also include applying a step function / activation function to the adjusted weighted sum. In some embodiments, the step function / activation function may be a nonlinear function. In various embodiments, such computations may include operations such as matrix multiplication. In some embodiments, the computations by multiple nodes may be performed in parallel, for example, using multiple processor cores of a multicore processor, individual processing units of a graphics processing unit (GPU), or dedicated neural circuits. In some embodiments, a node may include memory, which may, for example, be able to store and use one or more previous inputs when processing subsequent inputs. For example, a node with memory may include a Long Short-Term Memory (LSTM) node. An LSTM node may use memory to maintain “states” that enable the node to function like a finite state machine (FSM).

[0038] In some embodiments, the trained model may include embeddings or weights for individual nodes. For example, the model may start as a group of nodes organized into layers, as specified by the model format or structure. In initialization, the corresponding weights may be applied to the connections between nodes that are connected according to the model format, for example, the connections between each pair of nodes in a series of layers of a neural network. For example, each weight may be assigned randomly or initialized to a default value. The model may then be trained, for example, using training data to produce results.

[0039] Training may involve applying supervised learning techniques. In supervised learning, training data can include multiple inputs (e.g., multiple input panning videos) and corresponding ground truth outputs for each input (e.g., a ground truth panoramic video for each input panning video). Based on a comparison between the model's output (e.g., panoramic video) and the ground truth output (e.g., ground truth panoramic video), the weight values ​​are automatically adjusted, for example, in a way that increases the probability that the model generates a ground truth panoramic video.

[0040] In various embodiments, the trained model includes a set of weights, or embeddings, corresponding to the model structure. In some embodiments, the trained generative machine learning model may include an initial set of weights, for example, downloaded from a server that provides the weights. In various embodiments, the trained generative machine learning model includes a set of weights, or embeddings, corresponding to the model structure. In embodiments where data is omitted, the media application 103 may generate a trained generative machine learning model based on prior training, for example, by the developer of the media application 103, by a third party, etc.

[0041] In some embodiments, if the generative machine learning model includes a convolutional neural network trained using supervised learning, training the generative machine learning model may include acquiring a panoramic video based on the input panning video for each input panning video. The generative machine learning model may calculate a loss value based on a comparison of the panoramic video with a ground truth panoramic video (included in the training data). For example, the generative machine learning model may calculate an optical flow endpoint error (EPE) to measure the consistency of the generated motion. The flow between consecutive frames of the ground truth video and the output video may be calculated as an L2 difference. In some embodiments, small motion at low image resolution may be evaluated based on grid-based flow to generate subpixel alignment that is more reliable than network-based flow. In some embodiments, pixel-level metrics may be evaluated separately for static and dynamic regions.

[0042] A generative machine learning model may update the weights of one or more nodes in a convolutional neural network based on the loss value (for example, after performing other cycles of tuning and training, so that the loss value decreases until it falls below a threshold).

[0043] In some embodiments, the generative machine learning model is a token-based model. Figure 4 shows an exemplary token-based model 400 that outputs panoramic video according to some embodiments described herein. The token-based model 400 includes an encoder 410, a bidirectional transformer 420, and a decoder 430. The encoder 410 receives a downsampled masked video 405 (e.g., a reprojected input panning video) (e.g., 160 x 96 pixels at 11 frames per second) and compresses the masked video 405 into masked tokens 415. In some embodiments, the mask includes input pixels for which synthesized pixels are generated relative to the area outside the mask. The masked tokens 415 represent discrete embeddings that describe the content of the masked video 405. In some embodiments, the attention layer of the encoder 410 is masked such that later frames pay attention to earlier frames in the window, but earlier frames cannot pay attention to later frames.

[0044] The bidirectional transformer 420 iteratively converts the masked tokens 415 to video tokens 425. The bidirectional transformer 420 converts the coarsest level clip (x K ) A forward and reverse pass is performed on the panoramic video to reduce or remove artifacts. The bidirectional transformer 420 may obtain the result of the forward pass in the region previously verified by the bidirectional transformer 420 and combine the result with the reverse pass in the rest of the region to obtain a video token 425. The video token 425 is decoded by the decoder 430 and outputs a similarly downsampled (e.g., 160 x 96 pixels at 11 frames per second) panoramic video 435. The panoramic video 435 is upsampled (e.g., 320 x 192 pixels).

[0045] In some embodiments, the generative machine learning model is a diffusion model, such as a spatiotemporal pixel diffusion model. The diffusion model is trained to generate a panoramic video by gradually adding noise to the panoramic video (noise addition) and then performing a denoising process to recover the original panoramic video from the noise. The media application 103 receives a reprojected input panning video as input and trains the diffusion model to output a panoramic video that includes synthesized pixels for the missing parts of the panoramic canvas.

[0046] In some embodiments, the media application 103 trains a diffusion model on an image outpainting task, where the training data includes image pairs of a portion of a ground truth image and a corresponding image from which random pixels or groups of pixels have been removed. As a result of training the diffusion model on the outpainting task, the diffusion model is trained to receive an incomplete input image and output an image with composite pixels that replace any missing areas.

[0047] In some embodiments, the diffusion model is fine-tuned using a training video in which a fixed mask is replaced with a panning mask to simulate a situation where the user captures input panning video by moving the camera left and right. In some embodiments, the media application 103 fine-tunes the trained diffusion model using a video pair of a natural video (ground truth video) paired with a masked video having a synthetic panoramic mask designed to mimic a real mask.

[0048] Figure 5 shows an exemplary diffuse model 500 trained to output panoramic video according to several embodiments described herein. The diffuse model 500 receives noise 505 and masked video 510. In some embodiments, the diffuse model 500 receives downsampled noise 505 and masked video 510 (e.g., 80 frames of 128 × 128 pixels) and outputs similarly downsampled panoramic video 520. The panoramic video 520 may be a base layer that is upsampled to 1024 × 1024 pixels.

[0049] Figures 6A–6D show exemplary processes for generating panoramic video. While these examples refer to a specific number of frames, the frame count is used as an example, and other frame counts may be used. Figure 6A includes process 600 with input panning video 605. Panoramic projection 610 projects the input panning video 605 onto a panoramic canvas (the most temporally detailed input frame x 0 (Starting from). Spatial downsampling 615 is performed on the downsampled reprojected input video 620a (temporal coarse input frame x). K This is performed to form (which terminates at). The downsampled reprojected input video 620a may be gradually downsampled to match the native height of the generative machine learning model 625. For example, each level of the reprojected input video 620 may be associated with 2X downsampling. Although four levels are illustrated, any number of levels (k) may be used, such as two to five levels, or more than five levels. In some embodiments, the time scale k(x k The input video of ) has a time scale k(m k ) has a corresponding mask.

[0050] Two different sliding spatial windows 621, 622 are used to provide portions of the reprojected base input video 620a to the generative machine learning model 625. The sliding spatial windows 621, 622 can be associated with dimensions that vary according to the type of generative machine learning model 625 used to generate the base video 630a. The sliding spatial windows 621, 622 advantageously enable the process 600 to be used with input videos 605 of various panorama widths.

[0051] The generative machine learning model 625 generates the base video 630a by performing spatial outpainting to generate synthesized pixels for the missing regions. The model predictions within the sliding spatial windows 631, 632 are averaged, and new samples are extracted from the average.

[0052] In an embodiment where the generative machine learning model 625 is a diffusion model, the projected panorama canvas is trimmed to each input window, and the missing regions outside the image frame are outpainted. The Gaussian distribution (represented by μ and Σ) for the pixel values can be averaged, and new samples can be extracted using a diffusion method such as a denoising probabilistic model (DDPM).

[0053] In an embodiment where the generative machine learning model 625 is a token-based model, spatial aggregation can be performed by averaging the probability distributions predicted for the tokens prior to sampling.

[0054] The generative machine learning model 625 gradually restores the temporal dimension of the input video 605 by gradually completing the panorama video at an upsampled temporal resolution. For each level k ∈ [K-1,0], the generative machine learning model 625 combines the reprojected input video 620(x k ) with a coarser level of the completed panorama video (y k+1 ) to generate a low-resolution panorama video (y k ) of the completed.

[0055]

number

[0056] During this process, diffuse sampling is constrained to maintain the pixels that are temporally upsampled and synthesized in odd-numbered frames, while the synthesized pixels are resynthesized in even-numbered frames. In the case of fast-moving video (e.g., skiers, kayakers, skateboarders, or any scene where the subject's movement satisfies a threshold rate of change between consecutive frames of video), the pixels that were temporally upsampled in odd-numbered frames may also need to be resynthesized. In that case, the full-frame mask is maintained in odd-numbered frames for a predetermined amount of the sampling schedule (e.g., 1 / 8), and then the original mask across the entire window (m) is restored so that the diffuse model can synthesize new temporal details in the missing areas outside the original mask. k ) is used.

[0057] During outpainting 720, in the time dimension, the diffusion model 734 is applied in a sliding window manner where half of the windows overlap between two windows 730 and 732 (e.g., 40 frames). In the spatial dimension, multiple overlapping predictions are computed in parallel by the diffusion model 734 and then aggregated (736) to complete the panoramic video (y k )725 is extracted from the average.

[0058] In some embodiments, the diffusion model 734 is fine-tuned by randomly trimming the video to 128x128 pixels and expanding the composite panning video mask and the actual set of panning video masks. In some embodiments, the diffusion model 734 is optimized for a diffusion denoising object with squared error loss using a dynamic mask (i.e., a mask that changes position in subsequent frames to simulate the input panning video).

[0059] Compared to diffusion models that perform outpainting, token-based models mask and regenerate corresponding tokens by masking (m k Outpaint the pixels outside of the ) area.

[0060]

number

[0061] In some embodiments, the token-based model is fine-tuned to reduce fidelity loss in the encoder / decoder architecture, better align the synthesized pixels with the input video, and preserve input detail in out-painted areas. The decoder may be fine-tuned on a patch of valid pixels before the final result is synthesized.

[0062] Referring to Figure 8, an exemplary comparison 800 of how the probabilities during outpainting using a token-based model in some embodiments described herein are comparable to those of a diffusion model is shown. The reprojected input video 801 includes missing regions 802. The generative machine learning model generates a sample 803 from two predicted probability distributions associated with two overlapping windows, namely the left window 804 and the right window 805. The token-based model outputs discrete token probabilities 806, 807 based on the left window 804 and the right window 805, respectively, which are aggregated 808. The diffusion model outputs pixel probabilities 809, 810 based on a Gaussian distribution for pixel values ​​(represented as μ and Σ), which are aggregated 811.

[0063] Once the final upsampled panoramic video 630d is generated, the spatial superresolution 635 is applied to the upsampled panoramic video 630d. The result is merged with the output from reprojecting the input panning video 605 to obtain the full-resolution panoramic video 645.

[0064] Figure 6B shows the exemplary number of frames at each stage of the generative machine learning model, which is a token-based model 656 accepting 11 frames 655. The input video 651 contains 35 frames. For the coarsest time scale, the reprojected input video is completed with 11 frames 655. 2X upsampling of 11 frames is 22 frames 654, 2X upsampling of 22 frames is 44 frames 653, and the 44-frame version is resampled to 35 frames 652.

[0065] Figure 6C shows an exemplary process 660 for converting an input video 661 into a panoramic video. The input video is a video of a person in a kayak. The dashed rectangles 662 within the input video highlight sections of the input video 661 where unobserved movement occurs during the capture of the panning video. For example, if one person's kayak moves from left to right and another person's kayak moves from right to left, the activity within the dashed rectangles 662 will not be visible to the camera.

[0066] One difficulty in generating panoramic video is ensuring that movements, such as those of a person using a paddle, are properly aligned. The reprojected input video 663 contains unknown regions 664 where portions of the reprojected input video 663 should be filled in with composite pixels to create the panoramic video. The movement of a person with a paddle in the reprojected input video 663 is from a standstill to starting to paddle and then stopping again. If the composite pixels generated by the generative machine learning model 665 are not accurate, various problems may occur instead of the expected smooth paddle movement (e.g., discontinuous, jerky, or unnatural movement of the paddle).

[0067] The generative machine learning model 665 outputs a panoramic video 666 in which missing regions are replaced with composite pixels and the composite image is merged with the reprojected input video 663. For example, original pixels in unmasked regions within the reprojected input video 663 are merged with the composite image.

[0068] Figure 6D shows process 670 for finishing a panoramic video using a token-based model 679. In this example, a low-resolution panoramic video 671 includes 11 frames with upsampled temporal context in a reprojected input video 672, which is then upsampled with the reprojected input video 672 containing 22 frames and temporal detail, resulting in a 2x upsampled video 673 with 22 frames. The upsampled video is modified using the process described in Figure 6A, for example, by performing spatial alignment 674 (e.g., using temporal upsampling and frame rate matching), color alignment 675 (e.g., through color histogram matching), and synthesis 676 (e.g., using gridwarp-based optical flow or other types of image interpolation and / or blending), resulting in a merged video 677 with 22 frames. In this example, a token-based model is used to generate the synthesized pixels. In some embodiments where different types of generative machine learning models are used, such as a diffusion model, color alignment may not be performed.

[0069] The token-based model 679 may receive a merged video 677 as input, having a mask 678 in which white areas represent the input and black areas represent the reconstructed pixels. The token-based model 679 may generate a mask of pixels in the missing regions, reconstruct the pixels in the missing regions of the panoramic canvas, and merge the mask with the reconstructed pixels. The token-based model 679 outputs a high-resolution panoramic video 680 having 22 frames (i.e., having a threshold number of pixels).

[0070] Once the panoramic video is completed, it can be provided to the user who captured the input panning video. In some embodiments, the panoramic video is displayed with an option to select panoramic images from the panoramic video. For example, the panoramic video may be displayed as a series of panoramic images, and the user interface may include an option to isolate one or more of the panoramic images.

[0071] Exemplary Method Figure 9 shows a flowchart of an exemplary method 900 for generating a panoramic video. Method 900 may be performed by the computing device 200 in Figure 2. In some embodiments, Method 900 is performed by a user device 115 and a media server 101, or partially by the user device 115 and partially by the media server 101.

[0072] Method 900 in Figure 9 may begin in block 902. In block 902, an input panning video is received, which includes image frames captured by a camera panning at least once from a first side to a second side. In some embodiments, the quality of the panoramic video improves as a function of the number of times the image frames captured by the camera pan from the first side to the second side and from the second side to the first side. In some embodiments, the input panning video includes one or more moving objects. Block 902 may be followed by block 904.

[0073] In block 904, the input panning video is reprojected onto the panoramic canvas, which includes image frames and missing areas. Block 906 may follow block 904.

[0074] In block 906, the generative machine learning model outputs a panoramic video containing the synthesized pixels of the missing regions of the panoramic canvas by generating a base video with the missing region pixels on a coarse time scale, performing temporal upsampling, merging the temporally upsampled base video with the panoramic canvas, and reconstructing the synthesized pixels of the missing regions to refine the base video.

[0075] Temporal upsampling may include temporally upsampling the base video and performing spatial and color alignment of the temporally upsampled base video with the panoramic canvas to obtain an aligned temporally upsampled base video. Merging the temporally upsampled base video with the panoramic canvas may include merging the aligned temporally upsampled base video with the panoramic canvas. Recombining may include generating a mask of pixels in the missing areas, regenerating composite pixels in the missing areas based on the mask, and merging the regenerated pixels with the composite pixels.

[0076] In some embodiments, in response to outputting a panoramic video, a super-resolution pass is applied to the panoramic video to increase the resolution of the panoramic video. In some embodiments, a panoramic image is selected from the panoramic video.

[0077] In the above description, many specific details are given for illustrative purposes to provide a complete understanding of this specification. However, it will be apparent to those skilled in the art that this disclosure can be implemented without these specific details. In some cases, structures and devices are shown in block diagram form to avoid obscuring this specification. For example, embodiments may be described above with reference primarily to user interfaces and specific hardware. However, embodiments can be applied to any type of computing device capable of receiving data and commands, and any peripheral device that provides services.

[0078] In addition to the above description, the user may be provided with controls that allow the user to make choices regarding whether and when the systems, programs, or functions described herein may allow the collection of user information (e.g., information about the user's social networks, social behavior, or activities, occupation, user preferences, or user's current location), and whether content or communications are sent to the user from the server. Furthermore, certain data may be processed in one or more ways so that personally identifiable information is removed before it is stored or used. For example, a user's identity may be processed so that personally identifiable information cannot be determined, or if location information is obtained (e.g., at the city, zip code, or state level), the user's geographic location may be generalized so that the user's specific location cannot be determined. Thus, the user may have control over what information is collected about them, how that information is used, and what information is provided to them.

[0079] Any reference in this specification to “some embodiments” or “some examples” means that certain features, structures, or characteristics described in relation to the embodiments or examples may be included in at least one embodiment of this specification. The phrase “in some embodiments” appearing in various places in this specification does not necessarily refer to the same embodiments.

[0080] Some parts of the detailed explanation above are presented in terms of algorithms and symbolic representations of operations on data bits in computer memory. These descriptions and representations of algorithms are means used by those skilled in the art to most effectively communicate the content of their work to others skilled in the art. Here, and also generally, an algorithm is considered to be a self-consistent set of steps that lead to a desired result. These steps are steps that require the physical manipulation of physical quantities. Usually, though not always necessary, these quantities take the form of electrical or magnetic data that can be stored, transferred, combined, compared, or otherwise manipulated. For reasons of general use, it is sometimes convenient to refer to these data as bits, values, elements, symbols, characters, terms, numbers, etc.

[0081] However, it should be recognized that all these terms and similar terms should correspond to appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise specified, as will become clear from the following discussions, throughout this specification, discussions using terms such as “process,” “calculate,” “calculate,” “determine,” or “display” refer to the actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical (electronic) quantities in the registers and memory of the computer system and convert it into other data similarly represented as physical quantities in the memory or registers of the computer system, or in other such information storage devices, transmission devices, or display devices.

[0082] Embodiments of this specification may also relate to a processor for performing one or more steps of the methods described above. The processor may be a dedicated processor that is selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-temporary computer-readable storage medium, including but not limited to any type of disk including an optical disk, ROM, CD-ROM, magnetic disk, RAM, EPROM, EEPROM, magnetic card or optical card, flash memory including a USB key with non-volatile memory, or any type of medium suitable for storing electronic instructions, each connected to a computer system bus.

[0083] The specification may take the form of several entirely hardware embodiments, several entirely software embodiments, or several embodiments that include both hardware and software elements. In some embodiments, this specification is executed by software including, but not limited to, firmware, resident software, and microcode.

[0084] Furthermore, the specification may take the form of a computer program product accessible from a computer-enabled medium or computer-readable medium, which provides program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-enabled medium or computer-readable medium may be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, instruction execution unit, or instruction execution device.

[0085] A data processing system suitable for storing or executing program code would include at least one processor directly or indirectly connected to memory elements via a system bus. The memory elements may include local memory used during the actual execution of the program code, bulk storage, and cache memory providing temporary storage for at least some of the program code to reduce the number of times the code must be retrieved from bulk storage during execution.

Claims

1. A method performed by a computer, The system receives an input panning video containing image frames captured by a camera that pans at least once from the first side to the second side, The method performed by the computer further includes reprojecting the input panning video onto a panoramic canvas, wherein the panoramic canvas includes the image frames and missing regions, The process includes outputting a panoramic video that includes synthesized pixels of the missing regions of the panoramic canvas using a generative machine learning model, wherein the generative machine learning model To generate a base video including the composite pixels of the missing region on a coarse time scale, A computer-based method for outputting a panoramic video by performing temporal upsampling, merging the temporally upsampled base video with input pixels from the panoramic canvas, and refining the base video by recombining the composite pixels in the missing regions.

2. The aforementioned time upsampling is The aforementioned base video is upsampled in time, A computer-based method according to claim 1, comprising performing spatial and color alignment of the temporally upsampled base video with respect to the panoramic canvas in order to obtain aligned temporally upsampled base video.

3. The method performed by a computer according to claim 2, wherein merging the temporally upsampled base video with the panoramic canvas includes merging the aligned temporally upsampled base video with the panoramic canvas.

4. The method performed by a computer according to claim 3, wherein the recombination comprises generating a mask of pixels in the missing region, regenerating the combined pixels in the missing region based on the mask, and merging the regenerated pixels with the combined pixels.

5. The computer-operated method according to claim 1, wherein the quality of the panoramic video is improved as a function of the number of times the image frames captured by the camera pan from the first side to the second side and from the second side to the first side.

6. The computer-based method according to claim 1, wherein the input panning video includes one or more moving objects.

7. A method performed by a computer according to claim 1, further comprising applying a super-resolution pass to the panoramic video in order to increase the resolution of the panoramic video in response to outputting the panoramic video.

8. A method performed by a computer according to any one of claims 1 to 7, further comprising selecting a panoramic image from the panoramic video.

9. A program that, when executed by one or more computers, stores instructions that cause the one or more computers to perform an action, wherein the action is: The system receives an input panning video containing image frames captured by a camera that pans at least once from the first side to the second side, The operation further includes reprojecting the input panning video onto a panoramic canvas, wherein the panoramic canvas includes the image frames and missing regions, and the operation further includes The process includes outputting a panoramic video that includes synthesized pixels of the missing regions of the panoramic canvas using a generative machine learning model, wherein the generative machine learning model To generate a base video including the composite pixels of the missing region on a coarse time scale, The process involves performing temporal upsampling, merging the temporally upsampled base video with input pixels from the panoramic canvas, and recombining the composite pixels in the missing regions to refine the base video. A program that outputs the panoramic video.

10. The aforementioned time upsampling is The aforementioned base video is upsampled in time, The program according to claim 9, comprising performing spatial alignment and color alignment of the temporally upsampled base video with respect to the panoramic canvas in order to obtain aligned temporally upsampled base video.

11. The program according to claim 10, wherein merging the temporally upsampled base video with the panoramic canvas includes merging the aligned temporally upsampled base video with the panoramic canvas.

12. The program according to claim 11, wherein the recombination includes generating a mask of pixels in the missing region, regenerating the combined pixels in the missing region based on the mask, and merging the regenerated pixels with the combined pixels.

13. The program according to claim 9, wherein the input panning video includes one or more moving objects.

14. The program according to claim 9, wherein the operation further comprises applying a super-resolution pass to the panoramic video in order to increase the resolution of the panoramic video in response to outputting the panoramic video.

15. The program according to any one of claims 9 to 14, wherein the operation further comprises selecting a panoramic image from the panoramic video.

16. It is a system, Processor and The system comprises a memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to perform an action, and the action is, The system receives an input panning video containing image frames captured by a camera that pans at least once from the first side to the second side, The operation further includes reprojecting the input panning video onto a panoramic canvas, wherein the panoramic canvas includes the image frames and missing regions, and the operation further includes The process includes outputting a panoramic video that includes pixels synthesized in the missing areas of the panoramic canvas using a generative machine learning model, wherein the generative machine learning model To generate a base video including the composite pixels of the missing region on a coarse time scale, A system that outputs a panoramic video by performing temporal upsampling, merging the temporally upsampled base video with input pixels from the panoramic canvas, and refining the base video by recombining the composite pixels in the missing regions.

17. The aforementioned time upsampling is The aforementioned base video is upsampled in time, The system according to claim 16, comprising performing spatial alignment and color alignment of the temporally upsampled base video with respect to the panoramic canvas in order to obtain aligned temporally upsampled base video.

18. The system according to claim 17, wherein merging the temporally upsampled base video with the panoramic canvas includes merging the aligned temporally upsampled base video with the panoramic canvas.

19. The system according to claim 18, wherein the recombination includes generating a mask of pixels in the missing region, regenerating the combined pixels in the missing region based on the mask, and merging the regenerated pixels with the combined pixels.

20. The system according to any one of claims 16 to 19, wherein the input panning video includes one or more moving objects.

Images