Aesthetically controllable stereoscopic transformation

Abstract

To provide a technique for performing stereo conversion.SOLUTION: This technique includes generating, via a first machine learning model, depth information associated with objects depicted in an input image. The technique also includes generating, via a second set of machine learning models, a disparity map for the input image based on the depth information. The technique further includes extracting, via the second set of machine learning models, feature data from the input image and warping the feature data based on the generated disparity map. The technique further includes generating, via the second set of machine learning models, an output image such that the input image and the output image, when viewed together, form a stereo pair of images. The functions and transformations used in various machine learning models are fully differentiable, allowing improved end-to-end training of the machine learning models.SELECTED DRAWING: Figure 2

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority to U.S. patent application Ser. No. 18 / 473,947, filed Sep. 25, 2023, entitled "ARTISTICALLY CONTROLLABLE STEREO CONVERSION," which in turn claims the benefit of priority to U.S. provisional patent application Ser. No. 63 / 503,931, filed May 23, 2023, entitled "ARTISTICALLY CONTROLLABLE ARTIFICIAL INTELLIGENCE STEREO CONVERSION." The subject matter of these related applications is incorporated herein by reference. [Technical Field]

[0002] TECHNICAL FIELD Embodiments of the present disclosure relate generally to machine learning and computer vision, and more particularly to techniques for aesthetically controllable stereoscopic transformations using artificial intelligence. [Background technology]

[0003] Stereoscopic conversion refers to the creation of stereoscopic video content from traditional monocular (non-stereoscopic) video sequences. For example, live action feature films are often not shot in stereoscopic format, but it may be desirable for a movie studio to produce a stereoscopic version of the film to distribute alongside a non-stereoscopic version of the film. This is accomplished through stereoscopic conversion performed as a post-production process. Additionally, movie studios often have a large amount of legacy content, and stereoscopic conversion allows the studio to present this legacy content to audiences in stereoscopic format.

[0004] Existing stereoscopic transformation techniques may use computer graphics (CG) warping models to perform stereoscopic transformation. These techniques first identify, for a target frame, the relative differences in perceived depth between various objects in the scene (referred to herein as "disparity values"). The techniques then use the disparity values ​​to apply inverse warping or optical flow to generate an output image from the input image.

[0005] One drawback of the above technique is that there is no mechanism for the user to provide additional information based on a known reference input frame. Examples of additional information include explicit disparity values ​​or value ranges for the frame, annotations of interest points or regions in the frame, or identification of regions in the frame as fixed background.

[0006] Other existing techniques use deep neural networks to generate a stereo pair of images from a single input image by first applying a model pre-trained to generate reasonable depth information for the provided image. The depth information is then used to synthesize the second image needed to form the stereo pair and also generate a predicted disparity map. Similar to the above techniques, one drawback of deep neural network techniques is that users lack aesthetic control over the inference-time process, either by providing explicit disparity values ​​or by otherwise annotating the input images. Furthermore, neural network-based techniques often use non-differentiable functions or transformations, either in the neural network model itself or in auxiliary functions and features, which impacts the efficiency of end-to-end training of the stereo transformation model. Summary of the Invention [Problem to be solved by the invention]

[0007] As can be seen, there is a need in the art for more effective techniques for AI stereoscopic conversion of video frames or sequences. [Means for solving the problem]

[0008] One embodiment of the present invention provides a technique for performing stereoscopic transformation. The technique includes generating a predicted disparity map for an input image based on one or more depth values ​​associated with pixels in the input image using one or more machine learning models. The technique also includes extracting feature data from the input image representing pixels in the input image and applying an offset to each pixel represented by the feature data based on the predicted disparity map. The technique further includes identifying multiple pixels represented by the feature data that are offset to the same pixel location and assigning a visibility value to each of the multiple pixels based on the predicted disparity map. The technique further includes modifying each of the multiple pixels based on the visibility value so that one pixel of the multiple pixels is visible and the other multiple pixels are hidden, and generating an output image based on the modified multiple pixels such that the input image and the output image form a stereoscopic pair of images. [Effects of the Invention]

[0009] One technical advantage of the disclosed technique over conventional techniques is that the functions and transformations used in the various stereoscopic transformation steps are fully differentiable, allowing for improved end-to-end training of the disclosed stereoscopic transformation model. Furthermore, the disclosed technique allows for arbitrary aesthetic control input from the user, including the option to consider additional inputs beyond the input frames. Furthermore, the disclosed technique incorporates predicted disparity from input frames for depth-aware splatting and inpainting, and uses disparity information from known reference input frames when generating and compositing corresponding output frames. The optional use of multiple input frames also allows the disclosed technique to improve temporal consistency across multiple predicted output video frames. These technical advantages provide one or more technical improvements over conventional approaches.

[0010] So that the above features of the various embodiments may be understood in detail, the inventive concepts, briefly described above, will now be described in more detail with reference to various embodiments, some of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings illustrate only typical embodiments of the inventive concepts and, therefore, should not be construed in any way as limiting the scope of the invention, as there may be other equally effective embodiments. [Brief explanation of the drawings]

[0011] [Figure 1] 1 illustrates a computer system configured to implement one or more aspects of various embodiments. [Figure 2] FIG. 2 is a more detailed diagram of the training engine of FIG. 1 in accordance with various embodiments. [Figure 3A] 2 is a more detailed diagram of the stereoscopic transformation engine of FIG. 1 in accordance with various embodiments. [Figure 3B] 2 is a more detailed diagram of the stereoscopic transformation engine of FIG. 1 in accordance with various embodiments. [Figure 4] 1 illustrates an example input image and the results of various warping and disparity estimation processes applied to the input image according to some embodiments. [Figure 5] 1 is a flowchart of method steps for training a deep learning model to perform stereoscopic transformations according to various embodiments. [Figure 6] 1 is a flowchart of method steps for performing a stereoscopic transformation on a single input image according to various embodiments. [Figure 7] 1 is a flowchart of method steps for performing a stereoscopic transformation on a single video frame as an input image using neighboring video frames as keyframes according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0012] In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to one skilled in the art that the concepts of the present invention may be practiced without one or more of these specific details.

[0013] System Overview 1 illustrates a computing device 100 configured to implement one or more aspects of various embodiments. In one embodiment, computing device 100 includes a desktop computer, a laptop computer, a smartphone, a personal digital assistant (PDA), a tablet computer, or any other type of computing device configured to accept input, process data, and optionally display images, suitable for implementing one or more embodiments. Computing device 100 is configured to execute a training engine 122 and a stereoscopic transformation engine 124 resident in memory 116.

[0014] However, the computing devices described herein are merely exemplary, and any other technically feasible configurations are within the scope of this disclosure. For example, multiple instances of the training engine 122 and the stereoscopic transformation engine 124 may run on a set of nodes of a distributed and / or cloud computing system to perform the functions of the computing device 100. In another example, the training engine 122 and / or the stereoscopic transformation engine 124 may run on different sets of hardware, different types of devices, or environments to adapt the training engine 122 and / or the stereoscopic transformation engine 124 to different use cases or applications. In a third example, the training engine 122 and the stereoscopic transformation engine 124 may run on different computing devices and / or different sets of computing devices.

[0015] In one embodiment, computing device 100 includes, but is not limited to, one or more processing units 102, an input / output (I / O) device interface 104 coupled to one or more input / output (I / O) devices 108, a memory 116, a storage unit 114, and an interconnect (bus) 112 connecting a network interface 106. Processing unit 102 may be any suitable processing unit implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an artificial intelligence (AI) accelerator, any other type of processing device, or a combination of different processing units, such as a CPU configured to operate in cooperation with a GPU. In general, processing unit 102 may be any technically feasible hardware unit capable of processing data and / or executing software applications. Furthermore, in the context of this disclosure, the computing elements depicted in computing device 100 may correspond to physical computing systems (e.g., systems in a data center) or may be virtual computing instances running within a computing cloud.

[0016] I / O devices 108 include devices capable of providing input, such as a keyboard, mouse, or touch-sensitive screen, as well as devices capable of providing output, such as a display device. Additionally, I / O devices 108 may include devices capable of both accepting input and providing output, such as a touch screen or a universal serial bus (USB) port. I / O devices 108 may be configured to accept various types of input from an end user (e.g., a designer) of computing device 100 and to provide various types of output, such as displayed digital images, digital video, or text, to the end user of computing device 100. In some embodiments, one or more of I / O devices 108 are configured to couple computing device 100 to network 110.

[0017] Network 110 is any technically feasible type of communications network that enables data exchange between computing device 100 and external entities or devices, such as web servers, other networked computing devices, etc. For example, network 110 may include a wide area network (WAN), a local area network (LAN), a wireless (WiFi) network, and / or the Internet.

[0018] The storage unit 114 includes non-volatile storage for applications and data and may include a fixed or removable disk drive, a flash memory device, a CD-ROM, a DVD-ROM, a Blu-Ray, an HD-DVD, or other magnetic, optical, or solid-state storage device. The training engine 122 and the stereoscopic transformation engine 124 may be stored in the storage unit 114 and loaded into the memory 116 at run time.

[0019] Memory 116 may include a random access memory (RAM) module, a flash memory unit, or any other type of memory unit, or a combination thereof. Processing unit 102, I / O device interface 104, and network interface 106 are configured to read data from and write data to memory 116. Memory 116 contains various software programs and application data associated with the software programs that may be executed by processing unit 102, including a training engine 122 and a stereoscopic transformation engine 124.

[0020] In some embodiments, the training engine 122 trains one or more machine learning models to perform stereoscopic transformations, which modify the content of images or video frames to generate corresponding output images that form stereoscopic pairs with the input images or video frames. The stereoscopic transformation engine 124 executes the machine learning models to generate predicted output images or frames from the provided images or video frames. Throughout this description, the terms "frame" and "video frame" are used interchangeably.

[0021] More specifically, the training engine 122 and the stereoscopic transformation engine 124 are configured to train and execute one or more machine learning models that perform stereoscopic transformations on input images or video frames. The input images or video frames optionally include additional information, such as annotated disparity estimates or camera position / pose information. If video frames are provided as input, the additional information may also include immediately adjacent video frames or nearby video frames within a predetermined temporal distance from the input video frame. The adjacent or nearby video frames may be used to inform the generation of expected output video frames to improve the perceptibility of depth in the stereoscopic pair formed by the input and output video frames.

[0022] Stereoscopic transformation Figure 2 illustrates the training engine 122 of Figure 1 in more detail, according to some embodiments. The training engine 122 trains a machine learning model 230 that modifies input images or video frames to produce expected output images 250. The expected output images 250 and the corresponding input images or video frames form a stereoscopic pair of images. When viewed together, the stereoscopic pair of images depicts a scene that provides the illusion of relative depth between various objects in the scene and between the objects and the background. As illustrated, the training engine 122 includes the machine learning model 230, training data 210, and a training data preprocessing engine 220.

[0023] The training data 210 may include computer-generated imagery (CGI) images 212. CGI images are two-dimensional images or video frames that depict objects and / or backgrounds in a virtual three-dimensional space. In various embodiments, each CGI image 212 further includes explicit position, size, and shape data that describes the objects and / or backgrounds depicted in the CGI image.

[0024] The training data 210 may also include a pre-existing three-dimensional (3D) movie sequence 214. The 3D movie sequence 214 includes a number of frames, each frame including a stereoscopic pair of images that, when viewed together, depict a scene that, as previously described, provides the illusion of relative depth between various objects in the scene and between the objects and the background.

[0025] The training data preprocessing engine 220 processes the training data 210 and then provides the processed training data to the machine learning model 230. For example, in various embodiments, the training data preprocessing engine 220 processes the CGI image 212 and associated explicit position, size, and shape data to calculate relative depths between various objects depicted in the CGI image 212 and / or the background. The training data preprocessing engine 220 calculates ground truth disparity values ​​for objects depicted in the scene based on these calculated relative depths. In various embodiments, disparity refers to the offset, in pixels, between the location of an object depicted in one image of a stereoscopic pair (e.g., the left image) and the location of the same object depicted in the other image of the stereoscopic pair (e.g., the right image). This offset in the location of the object between the two images of the stereoscopic pair provides a desired illusion of depth when a viewer views the stereoscopic pair together.

[0026] In various embodiments, the training data 210 includes pre-existing 3D movie sequences 214. These pre-existing 3D movie sequences 214 each include a sequence of frames, with each frame represented as a stereoscopic pair of images. In various embodiments, not all stereoscopic paired images included in the pre-existing 3D movie sequences 214 are useful as training data for the stereoscopic transformation machine learning model. In such embodiments, the training data pre-processing engine 220 removes undesirable frames or frame sequences from the 3D movie sequences 214. For example, the training data pre-processing engine 220 removes frames from the pre-existing 3D movie sequences 214 that contain little or no useful information, such as black or blank frames. The training data pre-processing engine 220 may also remove frame sequences from the pre-existing 3D movie sequences 214 that do not have 3D effects applied, such as studio intros or producer credits. In frames without 3D effects applied, both images of a stereoscopic image pair will be identical and may not yield useful training information for the machine learning model 230.

[0027] The training data pre-processing engine 220 may also perform disparity estimation for frames of the existing 3D movie sequence 214. For a frame that comprises an image of a stereoscopic pair, the training data pre-processing engine 220 estimates a disparity value for that frame based on the relative difference between the position of an object in one image of the stereoscopic pair for that frame compared to the position of the same object in the other image of the stereoscopic pair.

[0028] Each existing 3D movie sequence 214 may contain hundreds of thousands of individual frames. In various embodiments, the training data preprocessing engine 220 divides the 3D movie sequence into multiple shots, each of which may contain a consecutive sequence of interrelated frames. For example, one shot may be a conversation scene between two characters, while the next shot may depict a car chase through a city. The training data preprocessing engine 220 may use an existing scene selection tool to divide the existing 3D movie sequence 214 into multiple shots, or the existing 3D movie sequence 214 may contain a list of editing decisions, such as timestamps or frame numbers, that delineate the beginning and end of individual shots within the 3D movie sequence.

[0029] After the training data preprocessing engine 220 removes undesirable frames or frame sequences and divides the 3D movie sequence into shots, the training data preprocessing engine 220 selects representative frames or frame sequences from the shots and further filters the shots to remove unsuitable frames before using them as training data. An unsuitable frame may have an average luminance that does not fall between a predetermined minimum and maximum level. Another example of an unsuitable frame is when a large number of pixels exhibit vertical disparity, i.e., when the representation of a particular object in one image of a stereoscopic pair is vertically offset compared to the representation of that object in the other image of the stereoscopic pair. Unlike horizontal disparity offset, vertical disparity offset does not contribute to the illusion of depth perceived in a stereoscopic pair and may indicate an insufficient 3D effect or insufficient disparity estimation. Similarly, a frame containing a high percentage of pixels with horizontal disparity below a predetermined threshold (e.g., 5 pixels) may indicate minimal applied 3D effect based on the disparity value estimated for that frame.

[0030] In some embodiments in which the training engine 122 trains the machine learning model 230 to perform stereoscopic transformations on images, the training data preprocessing engine 220 may select representative frames from a shot after filtering out irrelevant frames, as previously described. In other embodiments in which the training engine 122 trains the machine learning model 230 to perform stereoscopic transformations on image sequences, the training data preprocessing engine 220 may select representative frame sequences from a particular shot after further filtering out frame sequences that exhibit little or no object movement (i.e., still shots). For example, the training data preprocessing engine 220 may set a minimum object motion threshold between adjacent frames. In one non-limiting example, the minimum object motion threshold may be 8 pixels between adjacent frames.

[0031] The machine learning model 230 includes a disparity estimation network 235. The disparity estimation network 235 receives as input preprocessed training data, which may be either a CGI image or a frame of a representative frame sequence from an existing 3D movie sequence. In various embodiments where the input image to the disparity estimation network 235 is a frame of a representative frame sequence from an existing 3D movie sequence, the disparity estimation network 235 creates a reduced-resolution representation of an image from that frame (e.g., the left image or the right image) and predicts the depth d of each object and background in that image. The depth d represents the distance between the camera and the object or background shown in the image. Because images often depict background objects, such as the sun or sky, at a distant or near-infinite distance, in various embodiments, the disparity estimation network 235 predicts the inverse of the depth (1 / d). Very large depth values ​​associated with distant objects can complicate the estimation calculation and introduce errors. By calculating the reciprocal depth for the object, very large numbers are avoided in the subsequent estimation calculation, since the reciprocal depth for very distant objects tends to approach zero. The resulting reciprocal depth estimates are recorded pixel by pixel using a reduced resolution image.

[0032] The disparity estimation network 235 aligns the per-pixel reciprocal depth information for the reduced-resolution images to produce disparities, i.e., per-pixel displacement vectors. This alignment step generates a horizontal displacement vector for each pixel that is proportional to the inverse of the pixel's estimated depth. In this way, pixels representing foreground (nearby) objects receive larger displacement vectors and exhibit larger disparity values ​​than pixels representing distant objects. These varying horizontal displacements create the 3D effect of perceived depth in the output image predicted by the machine learning model 230.

[0033] In various embodiments in which the inputs provided to the disparity estimation network 235 are CGI images associated with ground truth disparity values, the disparity estimation network 235 aligns the estimated disparity values ​​with the ground truth disparity values ​​using least squares alignment. In various other embodiments in which the inputs to the disparity estimation network 235 are representative frame sequences from an existing 3D movie sequence, the disparity estimation network 235 calculates, for each frame of the representative frame sequence, a ground truth disparity value from the corresponding left and right images and aligns the estimated disparity value with the calculated ground truth disparity value using least squares alignment.

[0034] In some embodiments, the disparity estimation network 235 optionally performs disparity estimation on a reduced-resolution version of the input. The input can be a CGI image or a frame of a representative frame sequence from an existing 3D movie sequence. If the input is a frame of a representative frame sequence from an existing 3D movie sequence, the disparity estimation network 235 performs disparity estimation on either the right or left image of that frame. To generate a high-resolution disparity estimate, the disparity estimation network 235 upsamples from the low-resolution disparity estimate and the high-resolution input. Specifically, the disparity estimation network 235 uses bilateral upsampling with a neighborhood attention mechanism. In bilateral upsampling, the low-resolution disparity estimate is resampled at a higher resolution to match the input using a bilateral filter that preserves object edges in the high-resolution disparity estimate and smooths flat regions in the high-resolution disparity estimate. The neighborhood attention mechanism prioritizes disparity values ​​from nearby pixels when generating a disparity value for a newly generated pixel in the high-resolution disparity estimate.

[0035] 2, the machine learning model 230 also includes a transformation network 240. The disparity values ​​estimated by the disparity estimation network 235 are training inputs to the transformation network 240 of the machine learning model 230 and serve as trainable weight parameters. To avoid excessively large weight values ​​in the transformation network 240, the disparity estimation network 235 of various embodiments normalizes the estimated disparity values. Minimum and maximum desired disparity values ​​are predetermined for the disparity estimation network 235, and the disparity estimation network 235 normalizes the generated disparity values ​​to this range by applying a linear operator.

[0036] In operation, the transformation network 240 performs feature extraction on the input image, warps the feature space representation of the input image by the image's estimated disparity value, performs splatting to assign multiple pixels in the input image to the same pixels in the warped image, and interpolates image regions where previously occluded background regions are exposed by the warping. The transformation network 240 combines the warped, splatted, and interpolated feature space representation of the input image into a predicted output image 250.

[0037] The input to transformation network 240 is an input image and a high-resolution disparity estimate associated with that image. In various embodiments, the input to transformation network 240 may be a left image, for which it generates a corresponding predicted right image, or may be a right image, for which it generates a corresponding predicted left image. In various other embodiments, the input may be a center image, for which it generates both left and right output images (e.g., sequentially). In various embodiments where the right image is generated from an input left image, or where the left image is generated from an input right image, the input image and the generated predicted output image form a stereo pair. In various other embodiments where it generates both a left image and a right image from an input center image, the generated left and right images form a stereo pair.

[0038] The transformation network 240 includes a feature extraction network 242. The feature extraction network 242 performs pyramidal feature extraction on the input image at various levels of resolution. For each level of resolution, the feature extraction network 242 generates a feature vector that represents the input image at that resolution. The feature vector generated for a particular resolution will be different from the feature vector generated for a different resolution. The multiple feature vectors collectively form a multi-resolution feature vector that represents the input image in a feature space, rather than the input image's native representation as a matrix of RGB pixels.

[0039] The transformation network 240 further includes a warping and splatting network 244. The warping and splatting network 244 warps the feature space representation of the input image by the per-pixel displacement vectors calculated by the disparity estimation network 235. Specifically, each pixel of the feature space representation of the input image is shifted by a displacement vector proportional to the disparity value estimated for that pixel in the high-resolution disparity estimation.

[0040] The warped feature space representation of the input image contains disoccluded regions, where previously hidden areas of the background are exposed due to the displacement of objects in the scene. These disoccluded regions are left unfilled by the warping process and are filled in during a subsequent inpainting step. The warped representation also contains occluded regions, where multiple pixels from the input image are displaced to the same location in the feature space representation. The warping and splatting network 244 addresses these occluded regions using a differentiable warping strategy that includes weights that determine the ordering of multiple warped pixels to the same location. This ordering determines which of the multiple displaced (i.e., splatted) pixels to the same location in feature space should remain visible. The weights are generated from the per-pixel disparity values ​​estimated by the disparity estimation network 235, as previously described. Because foreground objects have smaller estimated depth values ​​compared to background objects, the disparity estimation network 235 assigns larger disparity values ​​to pixels of foreground objects. Thus, the disparity value directly indicates which splatted pixels are foreground and should remain visible, and which are background and should remain hidden. Before using the disparity values ​​to determine the order of the splatted pixels, the warping and splatting network 244 normalizes the disparity values ​​to the range [-1, 1] and scales the normalized values ​​using a trainable parameter α.

[0041] The warping and splatting network 244 performs the warping and splatting process using the multi-resolution feature vectors at all levels. For each resolution level, the warping and splatting network 244 upsamples the estimated disparity values ​​by interpolation to match the particular resolution level.

[0042] The warping and splatting network 244 also warps the original input image at the original resolution and at scaled resolutions corresponding to all levels of the extracted feature pyramid, thereby providing a user-visible representation of the warping and splatting effect. The warping and splatting network 244 also generates a warping mask that specifies how many pixels are splatted at each location in the warped input image. The warping and splatting network 244 also generates a binary disocclusion mask that specifies which pixels in the warped input image represent disoccluded background regions that require inpainting.

[0043] The transformation network 240 includes an image synthesis network 246 that performs background-aware inpainting of disoccluded regions and tackles sharp edges and aliasing in occluded regions with synthesis refinement. Finally, the image synthesis network 246 generates a predicted output image 250.

[0044] Given the warped image features and disparities for multiple resolutions as input, the image synthesis network 246 performs background-aware inpainting and synthetic refinement. Inpainting refers to the completion of background regions that were previously hidden but are now disoccluded by one or more foreground objects. Synthetic refinement reduces aliasing and sharp edges at the new boundaries between foreground objects and background where previously exposed background is now covered and occluded by one or more foreground objects. The image synthesis network 246 performs background-aware inpainting by applying a neighborhood-based self-attention technique to all resolution layers using local multi-scale self-attention. In various embodiments, the image synthesis network 246 includes a single-headed attention layer with a predetermined neighborhood size. This neighborhood-based self-attention technique ensures that disoccluded regions are inpainted with content based only on local background regions. After warping, background-aware inpainting, and synthetic refinement, the image synthesis network 246 converts the multi-resolution feature space representation into a final predicted output image. In various embodiments, the image synthesis network 246 performs the transformation using a GridNet convolutional neural network architecture that includes 3 rows and 6 columns, with bilinear upsampling replacing transposed convolutions in the network. In various embodiments, all steps of the stereoscopic transformation are differentiable. As a result, various networks can be trained together end-to-end.

[0045] When training the machine learning model 230, the disparity estimation network 235 is first trained separately. Alternatively, the training engine 122 may use a pre-trained version of the disparity estimation network 235.

[0046] The feature extraction network 242 and the image synthesis network 246 are trained by training the entire machine learning model 230 end-to-end while holding the weights of the disparity estimation network 235 constant. Training is supervised, using either CGI images with ground truth disparity information or left-right image pairs from a stereoscopic video dataset with associated annotated disparity information. Supervised training optionally includes data augmentation, in which image pairs from the stereoscopic video dataset are flipped either horizontally or vertically (e.g., with a probability of 0.5 / 0.1, respectively) to generate additional left-right image pairs for training. If a left-right image pair from a stereoscopic video dataset contains only negative disparity information, an offset may be applied to the right image of the pair to obtain both negative and positive disparities. The offset is determined by the median of the ground truth image disparities plus a random number between -10 and 10.

[0047] The disparity estimation network 235 generates predicted disparities, which are then aligned to the left-right disparities of the dataset using a least-squares method. In some embodiments, training continues for 200,000 steps using a batch size of 4 and a learning rate of 1e-3.

[0048] In the final step, the training engine 122 trains the entire pipeline end-to-end without fixing the weights of the disparity estimation network 235. The training is not only supervised using left and right image pairs, but also uses the disparity information of the dataset as an intermediate training signal for the disparity estimation network 235. This intermediate training signal is used as a weighting term in the loss function and can be progressively faded out during training.

[0049] During training, the training engine 122 compares the frame loss 280 with the expected output image 250 TIFF0007762256000001.tif6114

[0050] and the ground truth right frame I rFor the pixel-by-pixel comparison of , we calculate a separate disocclusion loss 260 weight β for the disocclusion region D as the L1 loss function (Equation 1).

[0051]

number

[0052] Additionally, the training engine 122 calculates the disparity and perceptual loss 270. The perceptual loss L F represents the difference between the expected output image of high-level semantic features and the ground truth right frame of the training data. d denotes the L1 distance between the expected disparity values ​​from the disparity estimation network 235 and the known training dataset disparities. Thus, the total loss function is

[0053]

number

[0054] is.

[0055] FIG. 3A is a diagram illustrating the stereoscopic transformation engine 124 of FIG. 1 in more detail, according to some embodiments. In these embodiments, the stereoscopic transformation engine 124 modifies a single input image 300 (e.g., a still image or a single frame from a video sequence) to generate an expected output image 345. The input image 300 and the generated expected output image 345 form a stereoscopic pair of images. When viewed together, this stereoscopic pair depicts a scene that provides the illusion of relative depth between the various objects in the scene and between the objects and the background. In various embodiments, the input image may be a left image, for which a corresponding expected right image is generated, or may be a right image, for which a corresponding expected left image is generated. In various other embodiments, the input is a center image, for which both left and right output images are generated (e.g., sequentially).

[0056] The stereoscopic transformation engine 124 includes a depth predictor 305. The depth predictor 305 modifies the input image 300 to produce a reduced-resolution version of the input image. The depth predictor 305 also predicts the depth d of objects and background in the reduced-resolution version of the input image. The depth d represents the distance between the camera and the object or background shown in the reduced-resolution version of the input image. Because input images often depict background objects, such as the sun or sky, at a distant or near-infinite distance, in various embodiments, the depth predictor 305 predicts the inverse depth (1 / d). The resulting inverse depth estimate is recorded for each pixel using the reduced-resolution version of the input image.

[0057] The stereoscopic transformation engine 124 performs disparity estimation 310 on the reduced-resolution version of the input image and the reciprocal depth estimate associated with the reduced-resolution version of the input image. The disparity estimation 310 aligns the per-pixel reciprocal depth information for the reduced-resolution version of the input image to produce a disparity, i.e., a per-pixel displacement vector. This alignment step generates a horizontal displacement vector for each pixel that is proportional to the reciprocal of the pixel's estimated depth. In this manner, pixels representing foreground (nearby) objects receive larger horizontal displacement vectors, representing larger disparity values, than pixels representing distant objects. These different horizontal displacements create the perceived 3D effect of depth in the expected output image 345 when viewed together with the input image 300. To avoid excessively large values ​​in the stereoscopic transformation engine 124, various embodiments of the disparity estimation 310 normalize the estimated disparity values. The minimum and maximum desired disparity values ​​are predetermined, and the disparity estimator 310 normalizes the generated disparity values ​​to this range by applying a linear operator. In various other embodiments, the user provides the reference disparity to the disparity estimator 310 in the form of a manual annotation of one or more portions of the input image 300 selected by the user (e.g., using paint strokes, defining a bounding box, or manually selecting an object in the image). The disparity estimator 310 aligns the expected disparity to the given reference disparity value, for example, using a least-squares alignment technique.

[0058] Disparity upsampling 315 modifies a reduced-resolution version of the input image and the associated disparity estimate to generate a high-resolution predicted disparity map 320. Disparity upsampling 315 performs upsampling from the low-resolution disparity estimate and the high-resolution input image. Specifically, disparity upsampling 315 uses bilateral upsampling with a neighborhood attention mechanism. In bilateral upsampling, the low-resolution disparity estimate is resampled at a higher resolution to match the input image using a bilateral filter that preserves object edges in the high-resolution predicted disparity map 320 and smooths flat regions in the high-resolution predicted disparity map 320. The neighborhood attention mechanism prioritizes disparity values ​​from nearby pixels when generating disparity values ​​for newly generated pixels in the high-resolution predicted disparity map 320.

[0059] Feature Extraction 322 performs pyramidal feature extraction on the input image at various levels of resolution. For each level of resolution, Feature Extraction 322 generates a feature vector that represents the input image at that resolution. The feature vector generated for a particular resolution will be different from the feature vector generated for a different resolution. The multiple feature vectors collectively form a multi-resolution feature vector that represents the input image in a feature space, rather than the input image's native representation as a matrix of RGB pixels.

[0060] Depth-aware splatting 325 warps the feature space representation of the input image by the per-pixel displacement vectors in the expected disparity map 320. Specifically, each pixel in the feature space representation of the input image is shifted by a displacement vector proportional to the disparity value estimated for that pixel in the expected disparity map 320.

[0061] The feature space representation of the warped input image contains disoccluded regions, where previously hidden areas of the background are exposed due to the displacement of objects in the scene. These disoccluded regions are left unfilled by the warping process and are filled in during a subsequent inpainting step. The warped representation also contains occluded regions, where multiple pixels from the input image are displaced to the same location in the feature space representation. Depth-aware splatting 325 addresses these occluded regions using a differentiable warping strategy that includes weights that determine the ordering of multiple warped pixels to the same location. This ordering determines which of the multiple displaced (i.e., splatted) pixels to the same location in feature space should remain visible. The weights are generated from the per-pixel disparity values ​​in the predicted disparity map 320. Because foreground objects have smaller estimated depth values ​​compared to background objects, disparity estimation 310 assigns larger disparity values ​​to pixels of foreground objects. Thus, the disparity value directly indicates which splatted pixels are foreground and should remain visible, and which are background and should remain hidden. Before using the disparity values ​​to determine the order of the splatted pixels, depth-aware splatting 325 normalizes the disparity values ​​to the range [-1, 1] and scales the normalized values ​​using a parameter α determined during training.

[0062] Depth-aware splatting 325 applies the above warping and splatting process using the multi-resolution feature vectors of all levels. For each resolution level, depth-aware splatting 325 upsamples the estimated disparity values ​​by interpolation to match the particular resolution level.

[0063] Depth-Aware Splatting 325 also warps the original input image at its original resolution and at scaled resolutions corresponding to all levels of the extracted feature pyramid, thereby providing a user-visible representation of the warping and splatting effects. Depth-Aware Splatting 325 also generates a warping mask that specifies how many pixels are splatted at each location in the warped input image. Depth-Aware Splatting 325 also generates a binary disocclusion mask that specifies which pixels in the warped input image represent disoccluded background regions that require inpainting.

[0064] Given the warped image features and disparities for multiple resolutions, Image Synthesis 330 performs Inpainting 335 and Synthetic Refinement 340. Inpainting refers to the completion of newly disoccluded background regions that were previously hidden by one or more foreground objects. Synthetic refinement reduces aliasing and sharp edges at new boundaries between foreground objects and background where previously exposed background is now covered and occluded by one or more foreground objects. Inpainting 335 performs background-aware inpainting by applying a neighborhood-based self-attention technique to all resolution layers using local multi-scale self-attention. In various embodiments, Inpainting 335 includes a single-headed attention layer with a predetermined neighborhood size. This neighborhood-based self-attention technique ensures that disoccluded regions are inpainted with content based only on local background regions.

[0065] After warping, background-aware inpainting, and synthetic refinement, Image Synthesis 330 transforms the multi-resolution feature space representation into a final expected output image 345. In various embodiments, Image Synthesis 330 performs the transformation using a GridNet convolutional neural network architecture that includes 3 rows and 6 columns, with bilinear upsampling replacing transposed convolutions in the network.

[0066] 3B illustrates the stereoscopic transformation engine 124 of FIG. 1 in more detail, according to some embodiments. In these embodiments, the stereoscopic transformation engine 124 modifies an input image 350 (e.g., a single 2D video frame) and an input keyframe 352 to generate a predicted output image 395. The input image 350 and the generated predicted output image 395 form a stereoscopic pair of images. In various embodiments, the input image 350 may be a left image, for which a corresponding predicted right image is generated, or may be a right image, for which a corresponding predicted left image is generated. In various other embodiments, the input is a center image, for which both a left and a right output image are generated (e.g., sequentially).

[0067] The input keyframe 352 is a video frame that is temporally related to the input image 350 (i.e., the input keyframe 352 precedes or follows the input image 350 in the video frame sequence). In some embodiments, the input keyframe 352 is selected from a fixed set of neighboring video frames TIFF0007762256000004.tif8114

[0068] However, TIFF0007762256000005.tif7114

[0069] represents the input image 350. In other embodiments, the input keyframes 352 are frames at exponentially increasing distances from the input image 350, e.g., TIFF0007762256000006.tif8114

[0070] The input keyframes 352 provide temporal context for the input image 350 to improve the accuracy and stability of the predicted output image 395.

[0071] The stereoscopic transformation engine 124 performs a stereoscopic transformation on the video sequence by processing multiple sequential input images 350 together with selected keyframes 352 associated with each input image 350 .

[0072] The stereoscopic transformation engine 124 includes a depth predictor 355. The depth predictor 355 modifies the input image 350 to produce a reduced-resolution version of the input image. The depth predictor 355 further predicts the depth d of objects and the background in the reduced-resolution version of the input image. In various embodiments, the depth predictor 355 predicts the inverse depth (1 / d), as previously described with respect to FIG. 3A. The resulting inverse depth estimate is recorded for each pixel using the reduced-resolution version of the input image.

[0073] For any given frame (i.e., input image 350 and input keyframe 352), depth predictor 355 further generates a scene point cloud by analyzing changing and static object and background positions across multiple frames. Depth predictor 355 also estimates the camera pose and configuration. To estimate the camera pose and position, depth predictor 355 may use, for example, structure-from-motion techniques. For each time step, depth predictor 355 generates a sparse depth rendering of the point cloud represented by input image 350 and input keyframe 352. Depth predictor 355 generates a reduced-resolution representation of input image 350 and each keyframe 352 that includes inverse depth values ​​for each pixel.

[0074] The stereoscopic transformation engine 124 performs disparity estimation 360 on the reduced-resolution version of the input image and the reciprocal depth estimate associated with the reduced-resolution version of the input image. Similar to disparity estimation 310 above, disparity estimation 360 aligns the per-pixel reciprocal depth information for the reduced-resolution version of the input image to produce a normalized expected disparity value for each pixel. In various other embodiments, a user provides a reference disparity to disparity estimation 360 in the form of a manual annotation of one or more portions of the input image 350 selected by the user. Disparity estimation 360 aligns the expected disparity to the provided reference disparity value using, for example, a least-squares alignment technique.

[0075] As previously described for disparity upsampling 315, disparity upsampling 365 modifies a reduced resolution version of the input image and the associated disparity estimate to generate a high-resolution predicted disparity map 370. Disparity upsampling 365 upsamples from the low-resolution disparity estimate and the high-resolution input image to generate a high-resolution predicted disparity map 370.

[0076] The stereoscopic transformation engine 124 generates a predicted output image 395 from the input image 350. For example, the input image 350 may be a TIFF0007762256000007.tif6114

[0077] The expected output image 395 can be the corresponding right image TIFF0007762256000008.tif5114

[0078] where the input image 350 and the expected output image 395 may form a stereo pair. The stereo transformation engine 124 may also consider additional images (i.e., keyframes) from other time instances. In the following description, the stereo transformation engine 124 may consider a single additional image TIFF0007762256000009.tif6114

[0079] only considers 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 39, 30, 31,

[0080] The stereoscopic transformation engine 124 generates motion vectors in the motion vector generation 372 to generate motion vectors for the key frames (in this case, TIFF0007762256000010.tif7114

[0081] ) to generate a predicted output image 395 (in this case, TIFF0007762256000011.tif6114

[0082] In some embodiments, the motion vector generation 372 warps the forward vector to the keyframe TIFF0007762256000012.tif7114

[0083] and the input image TIFF0007762256000013.tif6114

[0084] The motion vector generator 372 generates a motion vector based on the forward optical flow between time instance t and time instance t−1 and the disparity value for time instance t from the predicted disparity map 370. The forward optical flow includes a vector for each pixel that represents the motion of an object depicted in the scene between time instance t−1 and time instance t. The motion vector generator 372 then reverse-warps the disparity value for time instance t to time instance t−1. The motion vector generator 372 then adds the warped disparity value to the optical flow vector to obtain TIFF0007762256000014.tif7114

[0085] from TIFF0007762256000015.tif6114

[0086] This produces a forward vector to

[0087] In another embodiment, the motion vector generator 372 generates a backward vector from the input frame TIFF0007762256000016.tif6114

[0088] and keyframes TIFF0007762256000017.tif7114

[0089] The motion vector generation 372 then generates a motion vector based on the backward optical flow between time instances t and t-1. The backward optical flow includes pixel-by-pixel vectors that represent the motion of objects depicted in the scene between time instances t and t-1. TIFF0007762256000018.tif6114

[0090] The motion vector generation 372 also generates reverse (right-to-left) disparity values ​​by running the stereoscopic transformation engine 124 on the input image as already described for Figure 3A to calculate the negative of the warped per-pixel disparity values ​​obtained after inpainting 335 of Figure 3A. The motion vector generation 372 then adds the warped right-to-left disparity values ​​to the reverse optical flow to produce a reverse vector.

[0091] In yet another embodiment, the motion vector generator 372 generates predicted vectors directly based on an estimate of the output image generated from the input image as previously described for Figure 3 A. Given the estimated output image, the motion vector generator 372 directly estimates the optical flow between the input image and the estimated output image and calculates a forward or backward vector.

[0092] Feature Extraction and Depth-Aware Splatting 375 performs pyramidal feature extraction on the input image 350 and each keyframe 352 at various levels of resolution. For each level of resolution, Feature Extraction and Depth-Aware Splatting 375 generates a feature vector that represents the input image or keyframe at that resolution. Together, multiple feature vectors form a multi-resolution feature vector that represents the input image or keyframe in feature space.

[0093] Feature Extraction and Depth-Aware Splatting 375 warps the feature space representation of the input image by the motion vectors generated by Motion Vector Generation 372. Specifically, each pixel in the feature space representation of the input image is shifted by the value of the generated motion vector. In various embodiments including multiple keyframes 352, motion vectors are generated between each keyframe 352 and the input image 350, as previously described. Feature Extraction and Depth-Aware Splatting 375 then warps the feature space representation of the input image based on each of these multiple motion vectors.

[0094] The warped feature space representation of the input image includes disoccluded regions that remain unfilled by the warping process and are filled in during a subsequent inpainting step. The warped feature space representation also includes occluded regions, where multiple pixels from the input image are displaced to the same location in the feature space representation. Feature Extraction and Depth-Aware Splatting 375 addresses these occluded regions using a differentiable warping strategy that includes weights that determine the ordering of the multiple warped pixels to the same location. The weights are generated from the per-pixel disparity values ​​of the expected disparity map 370 in the same manner as described above for Depth-Aware Splatting 325. This weighted ordering determines which of the multiple displaced (i.e., splatted) pixels to the same location in feature space should remain visible.

[0095] Feature Extraction and Depth-Aware Splatting 375 performs the above warping and splatting process using the multi-resolution feature vectors of all levels. For each resolution level, Feature Extraction and Depth-Aware Splatting 375 upsamples the estimated disparity values ​​by interpolation to match the particular resolution level.

[0096] Feature Extraction and Depth-Aware Splatting 375 also warps the original input image at the original resolution and at scaled resolutions corresponding to all levels of the extracted feature pyramid, thereby providing a user-visible representation of the depth-aware warping and splatting effect. Feature Extraction and Depth-Aware Splatting 375 also generates a warping mask that specifies how many pixels are splatted at each location in the warped input image. Feature Extraction and Depth-Aware Splatting 375 also generates a binary disocclusion mask that specifies which pixels in the warped input image represent disoccluded background regions that require inpainting.

[0097] Using the warped image features and disparities for multiple resolutions, image synthesis 380 performs inpainting 385 and synthetic refinement 390. Inpainting and synthetic refinement fill in previously hidden, newly disoccluded background regions and reduce aliasing and sharp edges at new boundaries between foreground objects and the background in occluded regions. Inpainting 385 performs background-aware inpainting by applying neighborhood-based self-attention techniques using local multi-scale self-attention, as previously described for inpainting 335. Furthermore, inpainting 385 may analyze input image 350 and keyframes 352 to identify background regions that were previously visible in one or more keyframes 352 but are occluded in input image 350, and use these previously visible regions to refine the inpaint. For example, a particular section of the background may be visible in keyframe 352 representing time instance t−1 but is occluded in input image 350 representing time instance t. Inpaint 385 may identify that particular section of the background as newly disoccluded in expected output image 395 and use background information from the keyframe representing time instance t-1 to more accurately inpaint the newly disoccluded region.

[0098] After warping, background-aware inpainting, and synthesis refinement, image synthesis 380 transforms the multi-resolution feature space representations of input images 350 and keyframes 352 into a final predicted output image 395. In various embodiments, image synthesis 380 performs the transformation using a GridNet convolutional neural network architecture that includes 3 rows and 6 columns, with bilinear upsampling replacing transposed convolutions in the network.

[0099] 4 shows an example input image and the results of various warping and disparity estimation applied to the input image according to some embodiments. The top left panel of FIG. 4 shows input image 400. In various embodiments, input image 400 is, for example, a still image or a single frame of a 2D video sequence. Input image 400 depicts an object in the foreground of the image, as well as a background that is partially occluded by the object.

[0100] The top right panel of Figure 4 shows the results of depth-aware splatting 325, as already described for Figure 3A. Depth-aware splatting 325 offsets each pixel in the feature space representation of the input image 400 by an amount proportional to the disparity value estimated for that pixel. The pixel offsets produced by depth-aware splatting 325 result in occlusions 404 and disocclusions 402. Depth-aware splatting 325 correctly places foreground pixels in front of newly occluded background pixels, ensuring that boundary edges between foreground objects and the background are not distorted. The boundaries may contain sharp edges and aliasing artifacts, which are addressed in the following synthesis refinement 340, as already described for Figure 3A. Disocclusion 402 correctly paints pixels representing newly exposed areas of the background. Inpainting 335, as already described, fills in the disoccluded areas using only background pixels.

[0101] The bottom left panel of FIG. 4 shows a disparity representation 406 of the expected disparity map 320. As already described with respect to FIG. 3A, each pixel in the disparity map 320 includes an estimated disparity value. Depth-aware splatting 325 warps each pixel of the feature space representation of the input image by an offset proportional to the disparity value associated with that pixel. In the disparity representation 406, lighter regions represent foreground pixels closer to the viewer and have larger estimated disparity values, while darker regions represent background pixels farther from the viewer and have smaller estimated disparity values. In various embodiments, the disparity values ​​in the disparity representation 406 may be depicted using any suitable annotation scheme, e.g., darker pixels represent larger estimated disparity values ​​and lighter pixels represent smaller estimated disparity values. The disparity representation 406 may also use different colors to represent different ranges of disparity values.

[0102] The bottom center panel of Figure 4 shows warping mask 408. As already described for Figure 3A, depth-aware splatting 325 generates warping mask 408. Warping mask 408 represents the newly occluded regions in warped input image 400 and also represents how many pixels are splatted into each location in the warped input image.

[0103] The bottom right panel of Figure 4 shows a binary disocclusion mask 410. Depth-aware splatting 325 generates a binary disocclusion mask 410 that records which pixels in the warped input image represent newly disoccluded background regions that are interpolated with background pixels as already described for inpainting 335.

[0104] 5 is a flowchart of method steps for training a deep learning model to perform stereoscopic transformations according to various embodiments. The method steps are described with reference to the systems of FIGS. 1-3, but one skilled in the art will understand that any system configured to perform the method steps in any order is within the scope of the present disclosure.

[0105] As shown, in operation 502, training engine 122 executes a first neural network to generate inverse depth values ​​associated with objects in an input image. In various embodiments, the input image is a still image or a single frame from a video sequence. Training engine 122 generates inverse depth values ​​associated with each pixel in the input image. Training engine 122 also generates disparity values ​​associated with each pixel in the input image.

[0106] In operation 504, the training engine 122 trains the parameters of the first neural network using backpropagation and a disparity loss L d The disparity loss L d denotes the L1 distance between the expected disparity values ​​from the first neural network and the known training dataset disparity values.

[0107] In operation 506, the training engine 122 compares the disparity loss from the first neural network to a first predetermined threshold. If the first predetermined threshold is not met, then the training engine 122 returns to operation 502 and continues training the first neural network until the first predetermined threshold is met.

[0108] If the predetermined threshold is met, then the training engine 122 proceeds to operation 508. In operation 508, the training engine 122 pre-trains a second set of neural networks while holding the trained weights of the first neural network constant. The training engine 122 performs supervised training on the second set of neural networks using a dataset of left and right image pairs and disparity values ​​associated with the image pairs. As described with respect to Figures 3A and 3B, the second set of neural networks performs feature extraction using the input images and disparity values ​​associated with the input images. The second set of neural networks further performs depth-aware splatting, background-aware inpainting, and synthetic refinement to generate expected output images.

[0109] In operation 510, the training engine 122 updates the parameters of the second set of neural networks using backpropagation. As previously described with respect to FIG. 2, the training engine 122 calculates the frame loss as an L1 loss function L for the pixel-by-pixel comparison of the expected output image and the ground truth right frame of the training data for the image pair. L1 Additionally, the training engine 122 calculates the disparity and perceptual loss. F represents the difference between the expected output image of high-level semantic features and the ground truth right frame of the training data. d denotes the L1 distance between the predicted disparity values ​​from the first neural network and the known training dataset disparities. F and the disparity loss L d Each of the vectors has trainable weight parameters (γ and δ, respectively). The total loss function is therefore given by equation (2), as already described for FIG. 2.

[0110]

number

[0111] In operation 512, training engine 122 compares the total loss function given by equation (2) with a second predetermined threshold. If the second predetermined threshold is not met, training engine 122 continues with the second set of neural networks until the second predetermined threshold is met.

[0112] In operation 514, the training engine 122 executes the first neural network and the second set of neural networks end-to-end. Unlike operation 508 above, the parameters of the first neural network are no longer fixed.

[0113] Next, in operation 516, the training engine 122 performs supervised training using the dataset of left and right image pairs along with ground truth disparity values ​​associated with the dataset image pairs. In some embodiments, these ground truth disparity values ​​may be used as intermediate training signals to train the first neural network. These intermediate training signals may be progressively faded out during training. All functions and transformations in the first neural network and the second set of neural networks are fully differentiable. Therefore, the training engine 122 trains both the first neural network and the second set of neural networks end-to-end using backpropagation. The training engine 122 generates frame, perceptual, and disparity losses, as well as a total loss function. These loss functions are the same as those described above for operation 510.

[0114] In operation 518, training engine 122 compares the total loss function determined in operation 516 with a third predetermined threshold. If the third predetermined threshold is not met, training engine 122 continues training the first neural network and the second set of neural networks until the third predetermined threshold is met, terminating end-to-end training.

[0115] 6 is a flowchart of method steps for performing a stereoscopic transformation on a single input image according to various embodiments. The method steps are described with respect to the systems of FIGS. 1-3, but one skilled in the art will recognize that any system configured to perform the method steps in any order is within the scope of the present disclosure.

[0116] As shown, in operation 602, the stereoscopic transformation engine 124 accepts an input image. In various embodiments, the input image may be a still image or a single frame from a video sequence. In various embodiments, the stereoscopic transformation engine also accepts additional user input representing reference disparity information in the form of manual annotations of one or more portions of the input image selected by a user (e.g., using paint strokes, defining a bounding box, or manually selecting an object in the image).

[0117] In operation 604, the stereoscopic transformation engine 124 generates a predicted disparity map for the input image. The stereoscopic transformation engine 124 converts the input image into a low-resolution representation of the input image and executes a first trained neural network to perform depth prediction for various objects and background depicted in the low-resolution representation. The stereoscopic transformation engine 124 then calculates and stores the inverse depth value associated with each pixel of the low-resolution representation of the input image. For each pixel, the stereoscopic transformation engine 124 estimates a disparity value proportional to the inverse depth value associated with the pixel. The estimated disparity values ​​are aligned to further user input, for example, by least-squares alignment, if available. These estimated disparity values ​​represent the horizontal offset that needs to be applied for each pixel in subsequent warping and splatting operations. The stereoscopic transformation engine 124 then upsamples the disparity values ​​associated with the low-resolution representation of the input image to the full resolution of the input image, as previously described with reference to FIG. 3A. The stereoscopic transformation engine 124 converts the upsampled disparity estimate into a full-resolution predicted disparity map for the input image, where each pixel in the disparity map has an associated disparity value, with pixels representing foreground objects having larger associated disparity values, while background objects have smaller associated disparity values.

[0118] In operation 606, the stereo transformation engine 124 performs pyramidal feature extraction on the input image at various levels of resolution. For each level of resolution, the stereo transformation engine 124 generates a vector representing the features of the input image at the selected resolution. The feature vectors generated for the various resolutions will be different from each other. The multiple feature vectors collectively form a multi-resolution feature vector that represents the input image in a feature space, rather than the input image's native representation as a matrix of RGB pixels.

[0119] Next, in operation 608, the stereo transformation engine 124 modifies the feature space representation of the input image by performing warping and depth-aware splatting on the extracted feature data using a second set of neural networks. Each pixel in the feature space representation is warped (offset) by an amount proportional to the disparity value associated with that pixel. Because pixels at various depths have different disparity values, multiple pixels may be warped to the same location in the modified feature space representation, a process described as splatting. Splatting occurs at boundaries between foreground and background objects, where pixels in both the foreground and background are warped to the same location. Based on the disparity value associated with each warped pixel, the stereo transformation engine 124 determines which splatted pixels should remain visible in the modified feature space representation. Pixels with relatively large associated disparity values ​​represent objects closer to the viewer and should remain visible. Splatted pixels with relatively small disparity values ​​represent background objects that are far from the observer and should remain hidden in the modified feature space representation.

[0120] In operation 610, the stereo transformation engine 124 further modifies the feature space representation by inpainting disoccluded regions that take into account the background. Disoccluded regions are regions of the input image that were previously occluded by one or more foreground objects but that have become visible due to the warping process. The stereo transformation engine 124 artificially inpaints these regions using information solely from the image background, as previously described with respect to FIG. 3A. The stereo transformation engine 124 also synthetically refines the splatted pixels to remove sharp edges and aliasing artifacts.

[0121] In operation 612, the stereo transformation engine 124 transforms the warped, splatted, and inpainted feature space representation into a predicted output image that, when viewed together with the input image, forms a stereo pair that provides the illusion of different relative depths for the various objects depicted in the images.

[0122] 7 is a flowchart of method steps for performing stereoscopic transformation on a single video frame as an input image using neighboring video frames as input keyframes, according to various embodiments. The method steps are described with respect to the systems of FIGS. 1-3, but one skilled in the art will recognize that any system configured to perform the method steps in any order is within the scope of the present disclosure.

[0123] As shown, in operation 702, the stereoscopic transformation engine 124 accepts a frame from a video sequence as an input image and also accepts one or more neighboring video frames as input keyframes. In various embodiments, the input image may be a left image for which a corresponding predicted right image is generated, or may be a right image for which a corresponding predicted left image is generated. In various other embodiments, the input is a center image for which both left and right output images are generated (e.g., sequentially).

[0124] An input keyframe is a video frame that is temporally related to the input image (i.e., the input keyframe precedes or follows the input image 350 in the video frame sequence). In some embodiments, the input keyframe is a video frame that is temporally related to the input image 350 in the video frame sequence. TIFF0007762256000020.tif8114

[0125] However, TIFF0007762256000021.tif7114

[0126] represents the input image. In other embodiments, the input keyframes are frames at exponentially increasing distances from the input image, e.g., TIFF0007762256000022.tif7114

[0127] The input keyframes provide temporal context for the input images to improve the accuracy and stability of the expected output image. The stereoscopic transformation engine 124 performs a stereoscopic transformation on a video sequence by processing multiple sequential input images, along with selected keyframes associated with each input image. In the following description, the stereoscopic transformation engine 124 performs a stereoscopic transformation on a video sequence by processing multiple sequential input images, along with selected keyframes associated with each input image. TIFF0007762256000023.tif6114

[0128] only considers 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 39, 30, 31,

[0129] In operation 704, the stereo transformation engine 124 generates a high-resolution predicted disparity map for the input image and for each input keyframe. The stereo transformation engine 124 performs depth prediction by first generating reduced-resolution versions of the input image and each input keyframe using a first neural network. Next, the stereo transformation engine 124 predicts the depth of various objects and background in the image. In various embodiments, the stereo transformation engine 124 predicts the inverse of the depth, 1 / d, as previously described with reference to FIG. 3B. The stereo transformation engine 124 also generates a scene point cloud by analyzing the positions of changing or stationary objects and background across the input frame and one or more input keyframes. The stereo transformation engine 124 estimates the camera pose and configuration, for example, using structure-from-motion techniques. For each time step, the stereo transformation engine 124 generates a sparse depth rendering of the point cloud represented by the input image and input keyframe. The resulting reciprocal depth estimates are then recoded pixel-by-pixel using a reduced resolution version of the input image and a reduced resolution version of each input frame.

[0130] The stereoscopic transformation engine 124 performs disparity estimation on a reduced-resolution version of the input image and a reciprocal depth estimate associated with each pixel of the reduced-resolution version of the input image. As previously described with respect to FIG. 3B , the stereoscopic transformation engine 124 aligns the per-pixel reciprocal depth information for the reduced-resolution version of the input image to generate a normalized expected disparity value for each pixel. In various other embodiments, a user provides a reference disparity to the stereoscopic transformation engine 124 in the form of a manual annotation of one or more portions of the input image 350 selected by the user. The stereoscopic transformation engine 124 aligns the expected disparity to the provided reference disparity value using, for example, a least-squares alignment technique. The stereoscopic transformation engine 124 similarly performs disparity estimation for each input keyframe. For the input image and each input keyframe, the stereoscopic transformation engine 124 upsamples the associated low-resolution disparity estimate. The stereoscopic transformation engine 124 generates, for the input image and each input keyframe, a predicted disparity map by associating the upsampled disparity estimate with the corresponding high-resolution input image or keyframe.

[0131] In operation 706, the stereoscopic transformation engine 124 generates motion vectors to map the input keyframe (in one embodiment, TIFF0007762256000024.tif7114

[0132] ) is the expected output image TIFF0007762256000025.tif6114

[0133] In some embodiments, the stereoscopic transformation engine 124 warps the forward vector to the input keyframe TIFF0007762256000026.tif6114

[0134] and the input image TIFF0007762256000027.tif6114

[0135] The stereo transformation engine 124 generates a disparity map based on the forward optical flow between time instance t and time instance t−1 and the disparity values ​​for time instance t from the predicted disparity map. The forward optical flow includes per-pixel vectors that represent the motion of objects depicted in the scene between time instance t−1 and time instance t. The stereo transformation engine 124 then reverse-warps the disparity values ​​for time instance t to time instance t−1. The stereo transformation engine 124 then adds the warped disparity values ​​to the optical flow vectors to obtain TIFF0007762256000028.tif6114

[0136] from TIFF0007762256000029.tif6114

[0137] This produces a forward vector to

[0138] In another embodiment, the stereoscopic transformation engine 124 converts the inverse vector into the TIFF0007762256000030.tif7114

[0139] and input keyframe TIFF0007762256000031.tif7114

[0140] The stereo transformation engine 124 then generates a motion vector based on the backward optical flow between time instances t and t-1. The backward optical flow includes pixel-by-pixel vectors that represent the motion of objects depicted in the scene between time instances t and t-1. The stereo transformation engine 124 then converts this optical flow into a motion vector based on the motion vector of the frame. TIFF0007762256000032.tif6114

[0141] The stereo transformation engine 124 also generates reverse (right-to-left) disparity values ​​by performing the stereo transformation engine 124 on the input image as already described for Figure 3A to calculate the negative of the warped per-pixel disparity values ​​obtained after inpainting 335 of Figure 3A. The stereo transformation engine 124 then adds the warped right-to-left disparity values ​​to the reverse optical flow to produce a reverse vector.

[0142] In yet another embodiment, the stereoscopic transformation engine 124 converts the output image generated from a single input image as previously described with respect to FIG. 3A. TIFF0007762256000033.tif6114

[0143] Generate a direct predicted vector based on the estimate of the output image TIFF0007762256000034.tif6114

[0144] Given the estimate of TIFF0007762256000035.tif6114

[0145] and TIFF0007762256000036.tif7114

[0146] The optical flow between the two images is directly estimated and a forward or backward vector is calculated.

[0147] In operation 708, the stereo transformation engine 124 performs pyramidal feature extraction on the input image and each input keyframe at various levels of resolution. For each level of resolution, the stereo transformation engine 124 generates a feature vector that represents the input image or input keyframe at that resolution. The multiple feature vectors collectively form a multi-resolution feature vector that represents the input image or input keyframe in feature space.

[0148] The stereoscopic transformation engine 124 executes a second set of trained neural networks to warp the feature space representation of the input image by the motion vectors generated as described above. Specifically, each pixel of the feature space representation of the input image is shifted by the value of the generated motion vector. In various embodiments including multiple input keyframes, a motion vector is generated between each input keyframe and the input image. The stereoscopic transformation engine 124 then warps the feature space representation of the input image based on each of these multiple motion vectors.

[0149] As already described with respect to FIG. 6 , multiple pixels may be warped (splatted) to the same location in the feature space representation of the input image. The stereo transformation engine 124 approaches these occluded pixels using a differentiable depth-aware warping strategy that includes weights that determine the ordering of the multiple warped pixels to the same location. This weighted ordering is determined from the estimated disparity values ​​for each splatted pixel and determines which pixels will remain visible in the expected output image. The stereo transformation engine 124 performs the warping and splatting process using a multi-resolution feature vector representation of all levels of the input image. For each resolution level, the stereo transformation engine 124 upsamples the estimated disparity values ​​by interpolation to match the particular resolution level.

[0150] In operation 710, the stereo transformation engine 124 performs background-aware inpainting and synthetic refinement using features and disparities of the warped images for multiple resolutions. The inpainting and synthetic refinement fill in previously hidden but newly disoccluded background regions and reduce aliasing and sharp edges at new boundaries between foreground objects and the background in occluded regions. The stereo transformation engine 124 performs background-aware inpainting using local multi-scale self-attention, as previously described with respect to FIG. 3A . Additionally, the stereo transformation engine 124 may analyze the input images and input keyframes to identify background regions that were previously visible in one or more input keyframes but are now occluded in the input images, and use these regions to refine the inpainting. For example, a particular section of the background may be visible in the input keyframe representing time instance t−1, but is occluded in the input image representing time instance t. The stereoscopic transformation engine 124 may determine that particular section of the background as newly disoccluded in the expected output image and use the background information from the input keyframe representing time instance t-1 to more accurately inpaint the newly disoccluded area.

[0151] In operation 712, after warping, background-aware inpainting, and synthetic refinement, the stereoscopic transformation engine 124 transforms the multi-resolution feature space representation of the input images and input keyframes into a final expected output image.

[0152] In essence, the stereo transformation engine processes input images of video content to generate corresponding output images, such that the input images and the corresponding output images form a stereoscopic representation of the content of the input images. All functions and transformations used by the stereo transformation engine are fully differentiable, facilitating end-to-end training of the underlying stereo transformation model by backpropagation. In operation, the stereo transformation engine performs disparity estimation on the input images by calculating the inverse per-pixel depth for a reduced-resolution version of the input frame and then performing an alignment step to produce disparity values, i.e., point-wise displacement vectors of pixels. When calculating disparity values ​​for the input images, a range of possible disparity values ​​is provided, and the calculated values ​​are normalized to fall within this range, or a reference disparity is used to adjust the calculated values. The reference disparity can be in the form of a ground truth disparity (used during training) or a user-proposed disparity value provided at inference time. The user may also provide further aesthetic control inputs, such as selecting important objects in the input frames, identifying points in the input frames that correspond to stationary background regions, or providing camera parameters or camera placement information. The calculated and adjusted / normalized disparity values ​​are then used to generate a high-resolution disparity estimate. The stereo transformation engine further extracts features from the input frames at various resolutions.

[0153] The stereo transformation engine performs feature warping, depth-aware splatting, and inpainting in feature space based on the extracted features and high-resolution disparity estimation. Splatting refers to mapping (warping) multiple pixels to the same pixel with their associated displacement vectors in the feature space representation of the input image. Splatting is depth-aware because the stereo transformation engine determines which splatted pixels should appear in the foreground of the predicted target frame based on the estimated disparity values. The stereo transformation engine performs background-aware inpainting on disoccluded regions of the feature space representation of the input image. These disoccluded regions are filled in using only pixels and textures from the image background. The stereo transformation engine generates a predicted output image from the warped, splatted, and inpainted feature space image representation.

[0154] The above process for generating a single expected output image from a single input frame can be extended to accept multiple input images and generate multiple expected output images. Information contained in multiple adjacent, nearby, or near-neighboring input images is used to guide the stereoscopic transformation process, allowing the stereoscopic transformation engine to increase temporal consistency across the multiple expected output images, improving the illusion of depth perceived by a viewer in a stereoscopic pair comprising the input and expected output images.

[0155] One technical advantage of the disclosed technique over the prior art is that the functions and transformations used in the various stereoscopic transformation steps are fully differentiable, allowing for improved end-to-end training of the disclosed stereoscopic transformation model. Furthermore, the disclosed technique allows for arbitrary aesthetic control input from the user, including the option to consider additional inputs beyond the input frame. Furthermore, the disclosed technique incorporates predicted disparity from input images for depth-aware splatting and inpainting, and uses disparity information from known reference input frames when generating and compositing corresponding output images. By using an optionally large number of input images, the disclosed technique also allows for increased temporal consistency across multiple predicted output video images. These technical advantages provide one or more technical improvements over prior approaches.

[0156] 1. In various embodiments, a computer-implemented method includes generating a predicted disparity map for an input image based on one or more depth values ​​associated with pixels in the input image using one or more machine learning models; extracting from the input image feature data representing pixels in the input image; applying an offset to each pixel represented by the feature data based on the predicted disparity map; identifying a plurality of offset pixels represented by the feature data that are offset at the same pixel location; assigning a visibility value to each of the plurality of offset pixels based on the predicted disparity map; modifying each of the plurality of offset pixels based on the visibility value so that one pixel of the plurality of offset pixels is visible and the other offset pixels are hidden; and generating an output image based on the feature data and the modified plurality of offset pixels so that the input image and the output image form a stereoscopic pair of images.

[0157] 2. The computer-implemented method of claim 1, wherein generating a predicted disparity map comprises generating a reduced resolution version of the input image; assigning per-pixel depth values ​​to pixels representing foreground and background objects depicted in the reduced resolution version of the input image; assigning per-pixel predicted disparity values ​​in the disparity map based on the depth values ​​so that pixels representing foreground objects in the predicted disparity map are assigned higher disparity values ​​than pixels representing background objects; and upsampling per-pixel disparity map values ​​associated with the reduced resolution version of the input image to the original resolution of the input image.

[0158] 3. The computer-implemented method of claim 1 or 2, further comprising accepting one or more user annotations of the input image, each user annotation including identification of a selected portion of the input image and a ground truth disparity value associated with the selected portion; and aligning the expected disparity values ​​for each pixel of the selected portion of the input image with the associated ground truth disparity values ​​for the selected portion of the input image.

[0159] 4. The computer-implemented method of any one of items 1 to 3, wherein each of the one or more user annotations of the input image includes at least one of a bounding box surrounding a portion of the input image, one or more brush strokes indicating a portion of the input image, or a manual selection of an object depicted in the input image.

[0160] 5. The computer-implemented method of any one of items 1 to 4, wherein the step of modifying each of the plurality of offset pixels includes the steps of comparing associated disparity map values ​​for the plurality of offset pixels, and rendering one of the plurality of offset pixels visible based on the one of the plurality of offset pixels having the highest disparity map value, while hiding a pixel at the same pixel location having a lower disparity map value.

[0161] 6. The computer-implemented method of any one of items 1 to 5, wherein all functions and transformations performed by the one or more machine learning models are totally differentiable.

[0162] 7. The computer-implemented method of any one of items 1 to 6, further comprising the steps of: generating a disocclusion mask representing pixel locations in the feature data where portions of the image background are disoccluded after applying an offset to each offset pixel represented by the extracted feature data; interpolating the pixel locations represented by the disocclusion mask with image data selected from the image background; and generating an output image based on the modified offset pixels and the pixel locations represented by the interpolated disocclusion mask.

[0163] 8. The computer-implemented method of any one of items 1 to 7, wherein the image data used to complete the pixel locations represented by the disocclusion mask is selected by applying a neighborhood-based self-attention technique.

[0164] 9. The computer-implemented method of any one of items 1 to 8, wherein the input image is a single frame of a video sequence, the method further comprising the steps of: accepting as input one or more keyframes, each keyframe comprising a frame from the video sequence located a temporal distance from the input image; generating a predicted disparity map for each keyframe based on one or more depth values ​​associated with pixels in the keyframe; creating motion vectors between the input image and each keyframe based on the predicted disparity map for each keyframe, the motion vectors describing changes in position of objects depicted in the keyframes and the input image; extracting keyframe feature data from each keyframe; and applying the motion vectors to the extracted keyframe feature data for each keyframe to derive a step of completing pixel locations represented by the disocclusion mask.

[0165] 10. The computer-implemented method of any one of items 1 to 9, wherein a plurality of keyframes are accepted as input, the plurality of keyframes being selected at exponentially increasing temporal distances from the input image.

[0166] 11. In various embodiments, one or more non-transitory computer-readable media store instructions that, when executed by one or more processors, cause the one or more processors to: generate a predicted disparity map for an input image based on one or more depth values ​​associated with pixels in the input image using one or more machine learning models; extract from the input image feature data representing pixels in the input image; apply an offset to each pixel represented by the feature data based on the predicted disparity map; identify a plurality of offset pixels represented by the feature data that are offset at the same pixel location; assign a visibility value to each of the plurality of offset pixels based on the predicted disparity map; modify each of the plurality of offset pixels based on the visibility value so that one pixel of the plurality of offset pixels is visible and the other offset pixels are hidden; and generate an output image based on the feature data and the modified plurality of offset pixels so that the input image and the output image form a stereoscopic pair of images.

[0167] 12. The one or more non-transitory computer-readable media of item 11, wherein the instructions further cause the one or more processing units to: generate a reduced resolution version of the input image; assign per-pixel depth values ​​to pixels representing foreground and background objects depicted in the reduced resolution version of the input image; assign predicted per-pixel disparity values ​​in a disparity map based on the depth values ​​so that pixels representing foreground objects in the predicted disparity map are assigned higher disparity values ​​than pixels representing background objects; and upsample per-pixel disparity map values ​​associated with the reduced resolution version of the input image to the original resolution of the input image.

[0168] 13. One or more non-transitory computer-readable media described in item 11 or 12, wherein the instructions further cause the one or more processing units to perform the steps of accepting one or more user annotations for the input image, each user annotation including identification of a selected portion of the input image and a ground truth disparity value associated with the selected portion, and aligning the predicted disparity values ​​for each pixel for the selected portion of the input image with the associated ground truth disparity value for the selected portion of the input image.

[0169] 14. One or more non-transitory computer-readable media described in any one of items 11 to 13, wherein the instructions for performing the step of modifying each of the plurality of offset pixels further cause the one or more processing units to perform the steps of comparing associated disparity map values ​​for the plurality of offset pixels and rendering one pixel of the plurality of offset pixels visible based on the one pixel of the plurality of offset pixels having the highest disparity map value, while hiding a pixel at the same pixel location having a lower disparity map value.

[0170] 15. One or more non-transitory computer-readable media described in any one of items 11 to 14, wherein the instructions further cause the one or more processing units to perform the steps of generating a disocclusion mask representing pixel locations in the feature data where portions of the image background are disoccluded after applying an offset to each offset pixel represented by the extracted feature data, complementing the pixel locations represented by the disocclusion mask with image data selected from the image background, and generating an output image based on the modified plurality of offset pixels and the pixel locations represented by the complemented disocclusion mask.

[0171] 16. One or more non-transitory computer-readable media according to any one of items 11 to 15, wherein the input image is a single frame of a video sequence, and the instructions further cause the one or more processing units to: accept as input one or more keyframes, each keyframe comprising a frame from the video sequence located a temporal distance from the input image; generate a predicted disparity map for each keyframe based on one or more depth values ​​associated with pixels in the keyframe; create motion vectors between the input image and each keyframe based on the predicted disparity map for each keyframe, the motion vectors describing changes in position of objects depicted in the keyframes and the input image; extract keyframe feature data from each keyframe; and apply the motion vectors to the extracted keyframe feature data for each keyframe to derive a step of completing pixel locations represented by a disocclusion mask.

[0172] 17. In various embodiments, a computer-implemented method includes running a first neural network to predict per-pixel depth and disparity values ​​associated with an object depicted in an input image; updating parameters of the first neural network based on a first calculated training loss associated with the predicted disparity values; repeatedly running the first neural network to update the parameters of the first neural network; terminating execution of the first neural network if the first calculated training loss is less than a first threshold; sequentially running the first neural network and a second set of neural networks while holding the parameters of the first neural network constant; updating parameters of the second set of neural networks based on a second calculated training loss associated with the predicted output image; and updating parameters of the second set of neural networks while updating the parameters of the second set of neural networks. The method includes the steps of initiating a first iterative sequential execution of the first neural network and the second set of neural networks; terminating the first iterative sequential execution of the first neural network and the second set of neural networks if the second calculated training loss is less than a second threshold; sequentially executing the first neural network and the second set of neural networks; updating the first neural network and the second set of neural networks based on a third calculated training loss associated with the expected output image; initiating a second iterative sequential execution of the first neural network and the second set of neural networks while updating parameters of the first neural network and the second set of neural networks; and terminating the second iterative sequential execution of the first neural network and the second set of neural networks if the third calculated training loss is less than a third threshold.

[0173] 18. The first calculated training loss is the disparity loss L, which indicates the distance L1 between the expected disparity values ​​from the first neural network and the known training dataset disparity values. dItem 18. The computer-implemented method according to Item 17, wherein

[0174] 19. The second calculated training loss is a) a frame loss L that represents the distance L1 determined from a pixel-by-pixel comparison of the expected output image and the ground truth image versus the training data. L1 and b) the perceptual loss L, which represents the difference between the predicted output image of the high-level semantic features and the ground truth image vs. the training data. F 19. The computer-implemented method of claim 17 or 18, comprising at least one of:

[0175] 20. The third calculated training loss is L = L L1 +γ·L F +δ L d where γ and δ are the perceptual losses L, respectively. F and the disparity loss L d 20. The computer-implemented method of any one of items 17 to 19, wherein the weight parameters associated with

[0176] Any and all combinations of any claim element recited in any claim and / or any element described herein, in any aspect, are within the intended scope of the invention and protection.

[0177] The description of various embodiments is provided by way of example and is not intended to be exhaustive or to limit the scope to the disclosed embodiments. Numerous modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

[0178] Aspects of embodiments of the present invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware, all of which may be generally referred to herein as a "module," "system," or "computer." Furthermore, any hardware and / or software technique, process, function, component, engine, module, or system described in this disclosure may be implemented as a circuit or set of circuits. Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code thereon.

[0179] Any combination of one or more computer-readable media may be used. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples (non-exclusive list) of computer-readable storage media include an electrical connection including one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the context of this specification, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0180] Aspects of the present disclosure are described in terms of flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine. The instructions, when executed by the computer or other programmable data processing apparatus, cause the computer to perform the functions / acts indicated in the flowchart and / or block diagram blocks. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, an application-specific processor, or a field programmable gate array.

[0181] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams represents a module, segment, or portion of code, which may include one or more executable instructions that perform a particular logical function. It should also be noted that in some alternative embodiments, the functions shown in the blocks may be performed in a different order than that shown in the figures. For example, depending on the functionality involved, two successively shown blocks may in fact be executed substantially concurrently, or in some cases, the blocks may be executed in the reverse order. It will also be understood that each block of the block diagrams and / or flowcharts, and combinations of multiple blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system or a combination of dedicated hardware and computer instructions that performs a particular function or act.

[0182] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, the scope of which is defined by the appended claims. Preferred embodiments of the present invention will be described below in detail. Embodiment 1 1. A computer-implemented method for performing stereoscopic transformation, comprising: generating a predicted disparity map for an input image based on one or more depth values ​​associated with pixels in the input image using one or more machine learning models; extracting feature data from the input image that represents the pixels in the input image; applying an offset to each pixel represented by the feature data based on the predicted disparity map; identifying a plurality of offset pixels represented by the feature data, the offset pixels being offset to the same pixel location; assigning a visibility value to each of the plurality of offset pixels based on the predicted disparity map; modifying each of the plurality of offset pixels based on the visibility value such that one pixel of the plurality of offset pixels is visible and other pixels of the plurality of offset pixels are hidden; generating an output image based on the feature data and the modified plurality of offset pixels such that the input image and the output image form a stereoscopic pair of images; A computer-based method comprising: Embodiment 2 The step of generating a predicted disparity map includes: generating a reduced resolution version of the input image; assigning per-pixel depth values ​​to pixels representing foreground and background objects depicted in the reduced resolution version of the input image; assigning a predicted disparity value for each pixel in the disparity map based on the depth value, such that pixels representing foreground objects in the predicted disparity map are assigned a higher disparity value than pixels representing background objects; upsampling the disparity map values ​​for each pixel associated with the reduced resolution version of the input image to the original resolution of the input image; 2. The computer-implemented method of embodiment 1, comprising: Embodiment 3 receiving one or more user annotations of the input image, each user annotation including an identification of a selected portion of the input image and a ground truth disparity value associated with the selected portion; aligning the predicted disparity values ​​for each pixel for the selected portion of the input image with the associated ground truth disparity values ​​for the selected portion of the input image; 3. The computer-implemented method of embodiment 2, further comprising: Embodiment 4 4. The computer-implemented method of claim 3, wherein each of the one or more user annotations of the input image comprises at least one of a bounding box surrounding a portion of the input image, one or more brush strokes indicating a portion of the input image, or a manual selection of an object depicted in the input image. Embodiment 5 The step of modifying each of the plurality of offset pixels comprises: comparing the disparity map values ​​associated with the offset pixels; rendering the one of the plurality of offset pixels visible based on the one of the plurality of offset pixels having the highest disparity map value while obscuring pixels at the same pixel location having lower disparity map values; 2. The computer-implemented method of embodiment 1, comprising: Embodiment 6 2. The computer-implemented method of embodiment 1, wherein all functions and transformations performed by said one or more machine learning models are fully differentiable. Embodiment 7 generating a disocclusion mask representing pixel locations in the extracted feature data where portions of the image background are disoccluded after applying the offset to each offset pixel represented by the extracted feature data; interpolating the pixel locations represented by the disocclusion mask with image data selected from the image background; generating the output image based on the modified plurality of offset pixels and pixel locations represented by the interpolated disocclusion mask; 2. The computer-implemented method of embodiment 1, further comprising: Embodiment 8 8. The computer-implemented method of embodiment 7, wherein the image data used to complete pixel locations represented by the disocclusion mask is selected by applying a neighborhood-based self-attention technique. Embodiment 9 The input image is a single frame of a video sequence, and the method further comprises: accepting as input one or more keyframes, each keyframe comprising a frame from the video sequence located at a temporal distance from the input image; generating a predicted disparity map for each said keyframe based on one or more depth values ​​associated with pixels in that keyframe; generating motion vectors between the input image and each of the key frames that describe changes in position of objects depicted in the key frames and the input image based on a predicted disparity map for each of the key frames; extracting keyframe feature data from each said keyframe; applying the motion vectors to the extracted keyframe feature data for each of the keyframes to derive interpolation of pixel locations represented by the disocclusion mask; 8. The computer-implemented method of embodiment 7, comprising: Embodiment 10 10. The computer-implemented method of embodiment 9, wherein a plurality of keyframes are accepted as input, the plurality of keyframes being selected at exponentially increasing temporal distances from the input image. Embodiment 11 One or more non-transitory computer-readable media that store instructions that, when executed by one or more processors, cause the one or more processors to: generating a predicted disparity map for an input image based on one or more depth values ​​associated with pixels in the input image using one or more machine learning models; extracting feature data from the input image that represents the pixels in the input image; applying an offset to each pixel represented by the feature data based on the predicted disparity map; identifying a plurality of offset pixels represented by the feature data, the offset pixels being offset to the same pixel location; assigning a visibility value to each of the plurality of offset pixels based on the predicted disparity map; modifying each of the plurality of offset pixels based on the visibility value such that one pixel of the plurality of offset pixels is visible and other pixels of the plurality of offset pixels are hidden; generating an output image based on the feature data and the modified plurality of offset pixels such that the input image and the output image form a stereoscopic pair of images; One or more non-transitory computer-readable media that cause Embodiment 12 The instructions may further cause the one or more processing units to: generating a reduced resolution version of the input image; assigning per-pixel depth values ​​to pixels representing foreground and background objects depicted in the reduced resolution version of the input image; assigning a predicted disparity value for each pixel in the disparity map based on the depth value, such that pixels representing foreground objects in the predicted disparity map are assigned a higher disparity value than pixels representing background objects; upsampling the disparity map values ​​for each pixel associated with the reduced resolution version of the input image to the original resolution of the input image; One or more non-transitory computer-readable media as described in embodiment 11, which cause the following to occur: Embodiment 13 The instructions may further cause the one or more processing units to: receiving one or more user annotations of the input image, each user annotation including an identification of a selected portion of the input image and a ground truth disparity value associated with the selected portion; aligning the predicted disparity values ​​for each pixel for the selected portion of the input image with the associated ground truth disparity values ​​for the selected portion of the input image; 13. One or more non-transitory computer-readable media as described in embodiment 12, which cause the following to occur: Embodiment 14 The instructions for modifying each of the plurality of offset pixels may further include causing the one or more processors to: comparing the associated disparity map values ​​for the plurality of offset pixels; rendering the one of the plurality of offset pixels visible based on the one of the plurality of offset pixels having the highest disparity map value while obscuring pixels at the same pixel location having lower disparity map values; One or more non-transitory computer-readable media as described in embodiment 11, which cause the following to occur: Embodiment 15 The instructions may further cause the one or more processing units to: generating a disocclusion mask representing pixel locations in the extracted feature data where portions of the image background are disoccluded after applying the offset to each offset pixel represented by the extracted feature data; interpolating the pixel locations represented by the disocclusion mask with image data selected from the image background; generating the output image based on the modified plurality of offset pixels and pixel locations represented by the interpolated disocclusion mask; One or more non-transitory computer-readable media as described in embodiment 11, which cause the following to occur: Embodiment 16 the input image is a single frame of a video sequence; The instructions may further cause the one or more processing units to: accepting as input one or more keyframes, each keyframe comprising a frame from the video sequence located at a temporal distance from the input image; generating a predicted disparity map for each said keyframe based on one or more depth values ​​associated with pixels in that keyframe; generating a motion vector between the input image and each of the key frames based on the predicted disparity map for each of the key frames, the motion vector describing a change in position of an object depicted in the key frames and the input image; extracting keyframe feature data from each said keyframe; applying the motion vectors to the extracted keyframe feature data for each of the keyframes to derive interpolation of pixel locations represented by the disocclusion mask; 16. One or more non-transitory computer-readable media as described in embodiment 15, which cause the following to occur: Embodiment 17 1. A computer-implemented method for training a machine learning model to perform a stereoscopic transformation, comprising: running a first neural network that predicts pixel-by-pixel depth and disparity values ​​associated with objects depicted in the input image; updating parameters of the first neural network based on a first calculated training loss associated with the expected disparity value; repeatedly running the first neural network to update the parameters of the first neural network; terminating execution of the first neural network if the first calculated training loss is less than a first threshold; sequentially running the first neural network and a second set of neural networks while holding the parameters of the first neural network constant; updating the second set of neural network parameters based on a second calculated training loss associated with the expected output image; commencing a first iterative sequential execution of the first neural network and the second set of neural networks while updating the parameters of the second set of neural networks; terminating the first iterative sequential execution of the first neural network and the second set of neural networks if the second calculated training loss is less than a second threshold; sequentially executing the first neural network and the second set of neural networks; updating the first neural network and the second set of neural networks based on a third calculated training loss associated with the expected output image; commencing a second iterative sequential execution of the first neural network and the second set of neural networks while updating the parameters of the first neural network and the second set of neural networks; terminating the second iterative sequential execution of the first neural network and the second set of neural networks if the third calculated training loss is less than a third threshold; A computer-based method comprising: Embodiment 18 The first calculated training loss is a disparity loss L that indicates the distance L1 between the expected disparity values ​​from the first neural network and the known training dataset disparity values. d 18. The computer-implemented method of embodiment 17, wherein: Embodiment 19 The second calculated training loss is a) a frame loss L representing the distance L1 determined from a pixel-by-pixel comparison of the expected output image with the ground truth image versus the training data; L1 and, b) A perceptual loss L representing the difference between the predicted output image and the ground truth image vs. training data for high-level semantic features. F and 20. The computer-implemented method of embodiment 18, comprising at least one of: Embodiment 20 The third calculated training loss is L=L L1 +γ·L F +δ L d where γ and δ are the perceptual losses L, respectively. F and the disparity loss L d 20. The computer-implemented method of embodiment 19, wherein the trainable weight parameters associated with [Explanation of symbols]

[0183] 100 Computing equipment 122 Training Engine 124 Stereoscopic Conversion Engine 220 Training Data Preprocessing Engine 230 Machine Learning Models 235 Disparity Estimation Network 240 Conversion Network 242 Feature Extraction Network 244 Warping and Splatting Network 246 Image Synthesis Network

Claims

1. 1. A computer-implemented method for performing stereoscopic transformation, comprising: generating a predicted disparity map for an input image based on one or more depth values ​​associated with pixels in the input image using one or more machine learning models; extracting feature data from the input image that represents the pixels in the input image; applying an offset to each pixel represented by the feature data based on the predicted disparity map; identifying a plurality of offset pixels represented by the feature data, the offset pixels being offset to the same pixel location; assigning a visibility value to each of the plurality of offset pixels based on the predicted disparity map; modifying each of the plurality of offset pixels based on the visibility value such that one pixel of the plurality of offset pixels is visible and other pixels of the plurality of offset pixels are hidden; generating an output image based on the feature data and the modified plurality of offset pixels such that the input image and the output image form a stereoscopic pair of images; A computer-based method comprising:

2. The step of generating a predicted disparity map includes: generating a reduced resolution version of the input image; assigning per-pixel depth values ​​to pixels representing foreground and background objects depicted in the reduced resolution version of the input image; assigning a predicted disparity value for each pixel in the disparity map based on the depth value, such that pixels representing foreground objects in the predicted disparity map are assigned a higher disparity value than pixels representing background objects; upsampling the disparity map values ​​for each pixel associated with the reduced resolution version of the input image to the original resolution of the input image; 2. The computer-implemented method of claim 1, comprising:

3. receiving one or more user annotations of the input image, each user annotation including an identification of a selected portion of the input image and a ground truth disparity value associated with the selected portion; aligning the predicted disparity values ​​for each pixel for the selected portion of the input image with the associated ground truth disparity values ​​for the selected portion of the input image; The computer-implemented method of claim 2 further comprising:

4. 4. The computer-implemented method of claim 3, wherein each of the one or more user annotations of the input image comprises at least one of a bounding box surrounding a portion of the input image, one or more brush strokes indicating a portion of the input image, or a manual selection of an object depicted in the input image.

5. The step of modifying each of the plurality of offset pixels comprises: comparing the disparity map values ​​associated with the offset pixels; rendering the one of the plurality of offset pixels as visible based on the one of the plurality of offset pixels having the highest disparity map value, while obscuring pixels at the same pixel location having lower disparity map values; 2. The computer-implemented method of claim 1, comprising:

6. 10. The computer-implemented method of claim 1, wherein all functions and transformations performed by the one or more machine learning models are fully differentiable.

7. generating a disocclusion mask representing pixel locations in the extracted feature data where portions of the image background are disoccluded after applying the offset to each offset pixel represented by the extracted feature data; interpolating the pixel locations represented by the disocclusion mask with image data selected from the image background; generating the output image based on the modified plurality of offset pixels and pixel locations represented by the interpolated disocclusion mask; The computer-implemented method of claim 1 further comprising:

8. 8. The computer-implemented method of claim 7, wherein the image data used to complete pixel locations represented by the disocclusion mask is selected by applying a neighborhood-based self-attention technique.

9. The input image is a single frame of a video sequence, and the method further comprises: accepting as input one or more keyframes, each keyframe comprising a frame from the video sequence located at a temporal distance from the input image; generating a predicted disparity map for each said keyframe based on one or more depth values ​​associated with pixels in that keyframe; generating motion vectors between the input image and each of the key frames that describe changes in position of objects depicted in the key frames and the input image based on a predicted disparity map for each of the key frames; extracting keyframe feature data from each said keyframe; applying the motion vectors to the extracted keyframe feature data for each of the keyframes to derive interpolation of pixel locations represented by the disocclusion mask; 8. The computer-implemented method of claim 7, comprising:

10. 10. The computer-implemented method of claim 9, wherein a plurality of keyframes are accepted as input, the plurality of keyframes being selected at exponentially increasing temporal distances from the input image.

11. One or more non-transitory computer-readable media that store instructions that, when executed by one or more processors, cause the one or more processors to: generating a predicted disparity map for an input image based on one or more depth values ​​associated with pixels in the input image using one or more machine learning models; extracting feature data from the input image that represents the pixels in the input image; applying an offset to each pixel represented by the feature data based on the predicted disparity map; identifying a plurality of offset pixels represented by the feature data, the offset pixels being offset to the same pixel location; assigning a visibility value to each of the plurality of offset pixels based on the predicted disparity map; modifying each of the plurality of offset pixels based on the visibility value such that one pixel of the plurality of offset pixels is visible and other pixels of the plurality of offset pixels are hidden; generating an output image based on the feature data and the modified plurality of offset pixels such that the input image and the output image form a stereoscopic pair of images; One or more non-transitory computer-readable media that cause the

12. The instructions may further cause the one or more processing units to: generating a reduced resolution version of the input image; assigning per-pixel depth values ​​to pixels representing foreground and background objects depicted in the reduced resolution version of the input image; assigning a predicted disparity value for each pixel in the disparity map based on the depth value, such that pixels representing foreground objects in the predicted disparity map are assigned a higher disparity value than pixels representing background objects; upsampling the disparity map values ​​for each pixel associated with the reduced resolution version of the input image to the original resolution of the input image; 12. The one or more non-transitory computer-readable media of claim 11,

13. The instructions may further cause the one or more processing units to: receiving one or more user annotations of the input image, each user annotation including an identification of a selected portion of the input image and a ground truth disparity value associated with the selected portion; aligning the predicted disparity values ​​for each pixel for the selected portion of the input image with the associated ground truth disparity values ​​for the selected portion of the input image; 13. The one or more non-transitory computer-readable media of claim 12,

14. The instructions for modifying each of the plurality of offset pixels may further include causing the one or more processors to: comparing the associated disparity map values ​​for the plurality of offset pixels; rendering the one of the plurality of offset pixels as visible based on the one of the plurality of offset pixels having the highest disparity map value, while obscuring pixels at the same pixel location having lower disparity map values; 12. The one or more non-transitory computer-readable media of claim 11,

15. The instructions may further cause the one or more processing units to: generating a disocclusion mask representing pixel locations in the extracted feature data where portions of the image background are disoccluded after applying the offset to each offset pixel represented by the extracted feature data; interpolating the pixel locations represented by the disocclusion mask with image data selected from the image background; generating the output image based on the modified plurality of offset pixels and pixel locations represented by the interpolated disocclusion mask; 12. The one or more non-transitory computer-readable media of claim 11,

16. the input image is a single frame of a video sequence; The instructions may further cause the one or more processing units to: accepting as input one or more keyframes, each keyframe comprising a frame from the video sequence located at a temporal distance from the input image; generating a predicted disparity map for each said keyframe based on one or more depth values ​​associated with pixels in that keyframe; generating a motion vector between the input image and each of the key frames based on the predicted disparity map for each of the key frames, the motion vector describing a change in position of an object depicted in the key frames and the input image; extracting keyframe feature data from each said keyframe; applying the motion vectors to the extracted keyframe feature data for each of the keyframes to derive interpolation of pixel locations represented by the disocclusion mask; 16. The one or more non-transitory computer-readable media of claim 15,

17. 1. A computer-implemented method for training a machine learning model to perform a stereoscopic transformation, comprising: running a first neural network that predicts pixel-by-pixel depth and disparity values ​​associated with objects depicted in the input image; updating parameters of the first neural network based on a first calculated training loss associated with the predicted disparity value; repeatedly running the first neural network to update the parameters of the first neural network; terminating execution of the first neural network if the first calculated training loss is less than a first threshold; sequentially running the first neural network and a second set of neural networks while holding the parameters of the first neural network constant; updating the second set of neural network parameters based on a second calculated training loss associated with the expected output image; commencing a first iterative sequential execution of the first neural network and the second set of neural networks while updating the parameters of the second set of neural networks; terminating the first iterative sequential execution of the first neural network and the second set of neural networks if the second calculated training loss is less than a second threshold; sequentially executing the first neural network and the second set of neural networks; updating the first neural network and the second set of neural networks based on a third calculated training loss associated with the expected output image; commencing a second iterative sequential execution of the first neural network and the second set of neural networks while updating the parameters of the first neural network and the second set of neural networks; terminating the second iterative sequential execution of the first neural network and the second set of neural networks if the third calculated training loss is less than a third threshold; A computer-based method comprising:

18. The first calculated training loss is a disparity loss L that indicates the distance L1 between the predicted disparity values ​​from the first neural network and the known training dataset disparity values. ｄ 18. The computer-implemented method of claim 17, wherein:

19. The second calculated training loss is a) a frame loss L representing the distance L1 determined from a pixel-by-pixel comparison of the expected output image with the ground truth image versus training data; Ｌ１ and, b) a perceptual loss L representing the difference between the expected output image and the ground truth image versus training data for high-level semantic features; Ｆ and 20. The computer-implemented method of claim 18, comprising at least one of:

20. The third calculated training loss is L=L Ｌ１ +γ・L Ｆ +d・L ｄ where γ and δ are the perceptual losses L, respectively. Ｆ and the parallax loss L ｄ 20. The computer-implemented method of claim 19, wherein the weight parameters are trainable and associated with

Images