Artistically controllable stereo conversion

Abstract

To provide a technique for performing stereo conversion.SOLUTION: This technique includes generating, via a first machine learning model, depth information associated with objects depicted in an input image. The technique also includes generating, via a second set of machine learning models, a disparity map for the input image based on the depth information. The technique further includes extracting, via the second set of machine learning models, feature data from the input image and warping the feature data based on the generated disparity map. The technique further includes generating, via the second set of machine learning models, an output image such that the input image and the output image, when viewed together, form a stereo pair of images. The functions and transformations used in various machine learning models are fully differentiable, allowing improved end-to-end training of the machine learning models.SELECTED DRAWING: Figure 2

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority to U.S. Patent Application No. 18 / 473,947, filed September 25, 2023, entitled "ARTISTICALLY CONTROLLABLE STEREO CONVERSION," which in turn claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 503,931, filed May 23, 2023, entitled "ARTISTICALLY CONTROLLABLE ARTIFICIAL INTELLIGENCE STEREO CONVERSION," the subject matter of which is incorporated herein by reference. [Technical field]

[0002] FIELD OF THE DISCLOSURE Embodiments of the present disclosure relate generally to machine learning and computer vision, and more specifically to techniques for performing aesthetically controllable stereoscopic transformations using artificial intelligence. [Background technology]

[0003] Stereoscopic conversion refers to the creation of stereoscopic video content from traditional monocular (non-stereoscopic) video sequences. For example, live action feature films are often not shot in stereoscopic, but it may be desirable for a movie studio to produce a stereoscopic version of the film to distribute along with a non-stereoscopic version of the film. This is accomplished through stereoscopic conversion performed as a post-production process. Additionally, movie studios often have a large amount of legacy content, and stereoscopic conversion allows the studio to present this legacy content to audiences in stereoscopic view.

[0004] Existing stereoscopic transformation techniques may use computer graphics (CG) warping models to perform stereoscopic transformation. These techniques first identify for a target frame the relative difference in perceived depth between various objects in a scene (referred to herein as "disparity values"). They then use the disparity values ​​to apply inverse warping or optical flow to generate an output image from the input image.

[0005] One drawback of the above technique is that there is no mechanism for a user to provide additional information based on a known reference input frame. Examples of additional information include explicit disparity values ​​or value ranges for the frame, annotations of interest points or regions in the frame, or identification of regions in the frame as fixed background.

[0006] Other existing techniques use deep neural networks to generate a stereo pair of images from a single input image by first applying a model pre-trained to generate reasonable depth information for the provided image. The depth information is used to synthesize the second image required to form the stereo pair, and also to generate a predicted disparity map. As with the above techniques, one drawback of deep neural network techniques is that users do not have aesthetic control over the inference-time process, either by providing explicit disparity values ​​or by otherwise annotating the input images. Furthermore, neural network-based techniques often use non-differentiable functions or transformations, either in the neural network model itself or in auxiliary functions and features, which impact the efficiency of end-to-end training of the stereo transformation model. Summary of the Invention [Problem to be solved by the invention]

[0007] As can be seen, there is a need in the art for more effective techniques for AI stereoscopic conversion of video frames or sequences. [Means for solving the problem]

[0008] One embodiment of the present invention illustrates a technique for performing a stereoscopic transformation. The technique includes generating a predicted disparity map for an input image based on one or more depth values ​​associated with pixels in the input image using one or more machine learning models. The technique also includes extracting feature data from the input image that represents pixels in the input image, and applying an offset to each pixel represented by the feature data based on the predicted disparity map. The technique further includes identifying a plurality of pixels represented by the feature data that are offset to the same pixel location, and assigning a visibility value to each of the plurality of pixels based on the predicted disparity map. The technique further includes modifying each of the plurality of pixels based on the visibility value such that one pixel of the plurality of pixels is visible and the other pixels of the plurality of pixels are hidden, and generating an output image based on the modified plurality of pixels such that the input image and the output image form a stereoscopic pair of images. Effect of the Invention

[0009] One technical advantage of the disclosed technique over conventional techniques is that the functions and transformations used in the various stereoscopic transformation steps are fully differentiable, allowing for improved end-to-end training of the disclosed stereoscopic transformation model. Additionally, the disclosed technique allows for arbitrary aesthetic control input from the user, and the option to consider additional inputs beyond the input frames. Additionally, the disclosed technique incorporates predicted disparity from input frames for depth-aware splatting and inpainting, and uses disparity information from known reference input frames when generating and compositing corresponding output frames. The optional use of multiple input frames also allows the disclosed technique to increase temporal consistency across multiple predicted output video frames. These technical advantages provide one or more technical improvements over conventional approaches.

[0010] So that the above features of the various embodiments may be understood in detail, the inventive concept, briefly described above, will now be described in more detail with reference to various embodiments, some of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings depict only typical embodiments of the inventive concept, and therefore should not be construed in any way as limiting the scope of the invention, as there are other equally effective embodiments. [Brief description of the drawings]

[0011] [Figure 1] 1 illustrates a computer system configured to implement one or more aspects of various embodiments. [Diagram 2] FIG. 2 is a more detailed diagram of the training engine of FIG. 1 in accordance with various embodiments. [Figure 3A] 2 is a more detailed diagram of the stereoscopic transformation engine of FIG. 1 in accordance with various embodiments. [Figure 3B] 2 is a more detailed diagram of the stereoscopic transformation engine of FIG. 1 in accordance with various embodiments. [Figure 4] 1 illustrates an example input image and the results of various warping and disparity estimation processes applied to the input image according to some embodiments. [Diagram 5] 1 is a flowchart of method steps for training a deep learning model to perform stereoscopic transformations in accordance with various embodiments. [Figure 6] 4 is a flowchart of method steps for performing a stereoscopic transformation on a single input image according to various embodiments. [Figure 7] 4 is a flowchart of method steps for performing stereoscopic transformation on a single video frame as an input image using neighboring video frames as key frames according to various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of various embodiments. However, it will be apparent to one of ordinary skill in the art that the inventive concept may be practiced without one or more of these specific details.

[0013] System Overview 1 illustrates a computing device 100 configured to implement one or more aspects of various embodiments. In one embodiment, computing device 100 includes a desktop computer, a laptop computer, a smartphone, a personal digital assistant (PDA), a tablet computer, or any other type of computing device configured to accept input, process data, and optionally display images, suitable for implementing one or more embodiments. Computing device 100 is configured to execute a training engine 122 and a stereoscopic transformation engine 124 resident in memory 116.

[0014] However, the computing devices described herein are merely examples, and any other technically feasible configurations are within the scope of the present disclosure. For example, multiple instances of the training engine 122 and the stereoscopic transformation engine 124 may run on a set of nodes of a distributed and / or cloud computing system to perform the functions of the computing device 100. In another example, the training engine 122 and / or the stereoscopic transformation engine 124 may run on different sets of hardware, different types of devices, or environments to adapt the training engine 122 and / or the stereoscopic transformation engine 124 to different use cases or applications. In a third example, the training engine 122 and the stereoscopic transformation engine 124 may run on different computing devices and / or different sets of computing devices.

[0015] In one embodiment, the computing device 100 includes, but is not limited to, one or more processing units 102, an input / output (I / O) device interface 104 coupled to one or more input / output (I / O) devices 108, a memory 116, a storage unit 114, and an interconnect (bus) 112 connecting the network interface 106. The processing unit 102 may be any suitable processing unit implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), an artificial intelligence (AI) accelerator, any other type of processing device, or a combination of different processing units, such as a CPU configured to operate in cooperation with a GPU. In general, the processing unit 102 may be any technically feasible hardware unit capable of processing data and / or executing software applications. Furthermore, in the context of this disclosure, the computing elements depicted in the computing device 100 may correspond to physical computing systems (e.g., systems in a data center) or may be virtual computing instances running in a computing cloud.

[0016] I / O devices 108 include devices capable of providing input, such as a keyboard, mouse, touch-sensitive screen, and also devices capable of providing output, such as a display device. Additionally, I / O devices 108 may include devices capable of both accepting input and providing output, such as a touch screen, a universal serial bus (USB) port, and the like. I / O devices 108 may be configured to accept various types of input from an end user (e.g., a designer) of computing device 100 and provide various types of output, such as displayed digital images, digital video, or text, to the end user of computing device 100. In some embodiments, one or more of I / O devices 108 are configured to couple computing device 100 to network 110.

[0017] Network 110 is any technically feasible type of communications network that enables the exchange of data between computing device 100 and external entities or devices, such as web servers, other networked computing devices, etc. For example, network 110 may include a wide area network (WAN), a local area network (LAN), a wireless (WiFi) network, and / or the Internet.

[0018] The storage 114 includes non-volatile storage for applications and data and may include fixed or removable disk drives, flash memory devices, CD-ROMs, DVD-ROMs, Blu-Ray, HD-DVDs, or other magnetic, optical, or solid-state storage devices. The training engine 122 and the stereoscopic transformation engine 124 may be stored in the storage 114 and loaded into the memory 116 at run time.

[0019] The memory 116 may include a random access memory (RAM) module, a flash memory section, or any other type of memory section, or a combination thereof. The processing unit 102, the I / O device interface 104, and the network interface 106 are configured to read data from and write data to the memory 116. The memory 116 includes various software programs and application data associated with the software programs that may be executed by the processing unit 102, including a training engine 122 and a stereoscopic transformation engine 124.

[0020] In some embodiments, the training engine 122 trains one or more machine learning models to perform stereoscopic transformation, in which the content of an image or video frame is modified to generate a corresponding output image that forms a stereoscopic pair with the input image or video frame. The stereoscopic transformation engine 124 executes the machine learning models to generate predicted output images or frames from the provided images or video frames. Throughout this description, the terms "frame" and "video frame" are used interchangeably.

[0021] More specifically, the training engine 122 and the stereoscopic transformation engine 124 are configured to train and execute one or more machine learning models that perform stereoscopic transformation on input images or video frames. The input images or video frames optionally include additional information, such as annotated disparity estimates or camera position / pose information. If a video frame is provided as input, the additional information may also include immediately adjacent video frames or nearby video frames within a predetermined temporal distance from the input video frame. The adjacent or nearby video frames may be used to inform the generation of a predicted output video frame to improve the perceptibility of depth in the stereoscopic pair formed by the input and output video frames.

[0022] Stereoscopic transformation 2 illustrates the training engine 122 of FIG. 1 in more detail, according to some embodiments. The training engine 122 trains a machine learning model 230 that modifies input images or video frames to produce expected output images 250. The expected output images 250 and the corresponding input images or video frames form a stereo pair of images that, when viewed together, depict a scene that provides the illusion of relative depth between various objects in the scene and between the objects and the background. As illustrated, the training engine 122 includes the machine learning model 230, training data 210, and a training data pre-processing engine 220.

[0023] The training data 210 may include computer generated imagery (CGI) images 212. CGI images are two-dimensional images or video frames that depict objects and / or backgrounds in a virtual three-dimensional space. In various embodiments, each CGI image 212 further includes explicit position, size, and shape data that describes the objects and / or backgrounds depicted in the CGI image.

[0024] The training data 210 may also include an existing three-dimensional (3D) movie sequence 214. The 3D movie sequence 214 includes a number of frames, each frame including a stereoscopic pair of images that, as previously described, when viewed together, depict a scene that provides the illusion of relative depth between various objects in the scene, and between the objects and the background.

[0025] The training data pre-processing engine 220 processes the training data 210 and then provides the processed training data to the machine learning model 230. For example, in various embodiments, the training data pre-processing engine 220 processes the CGI image 212 and associated explicit position, size, and shape data to calculate the relative depth between various objects depicted in the CGI image 212 and / or the background. The training data pre-processing engine 220 calculates ground truth disparity values ​​for the objects depicted in the scene based on these calculated relative depths. In various embodiments, disparity refers to the offset in pixels between the location where an object is depicted in one image of a stereoscopic pair (e.g., the left image) and the location where the same object is depicted in the other image of the stereoscopic pair (e.g., the right image). This offset in the location of the object between the two images of the stereoscopic pair provides a desired illusion of depth when a viewer views the stereoscopic pair together.

[0026] In various embodiments, the training data 210 includes pre-existing 3D movie sequences 214. These pre-existing 3D movie sequences 214 each include a sequence of frames, with each frame represented as a stereoscopic pair of images. In various embodiments, not all stereoscopic pair of images included in the pre-existing 3D movie sequences 214 are useful as training data for the stereoscopic transformation machine learning model. In such embodiments, the training data pre-processing engine 220 removes undesirable frames or frame sequences from the 3D movie sequences 214. For example, the training data pre-processing engine 220 removes frames from the pre-existing 3D movie sequences 214 that contain no or little useful information, such as black or blank frames. The training data pre-processing engine 220 may also remove frame sequences from the pre-existing 3D movie sequences 214 that do not have 3D effects applied, such as studio intros or producer credits. In frames that do not have 3D effects applied, both images of a stereoscopic image pair will be identical and may not yield useful training information for the machine learning model 230.

[0027] The training data pre-processing engine 220 may also perform disparity estimation for frames of the existing 3D movie sequence 214. For a frame that includes an image of a stereoscopic pair, the training data pre-processing engine 220 estimates a disparity value for that frame based on the relative difference between an object position in one image of the stereoscopic pair for that frame compared to the position of the same object in the other image of the stereoscopic pair.

[0028] The existing 3D movie sequences 214 may each include hundreds of thousands of individual frames. In various embodiments, the training data pre-processing engine 220 divides the 3D movie sequences into multiple shots, where each shot may include a consecutive sequence of correlated frames. For example, one shot may be a conversation scene between two characters, while the next shot may depict a car chase through a city. The training data pre-processing engine 220 may divide the existing 3D movie sequences 214 into multiple shots using existing scene selection tools, or the existing 3D movie sequences 214 may include a list of editing decisions, such as timestamps or frame numbers, that delineate the beginning and end of individual shots in the 3D movie sequence.

[0029] After the training data pre-processing engine 220 removes undesirable frames or frame sequences and divides the 3D movie sequence into shots, the training data pre-processing engine 220 selects representative frames or frame sequences from the shots and further filters the shots to remove unsuitable frames before using them as training data. Unsuitable frames may have average luminance that does not fall between a predetermined minimum and maximum level. Another example of an indication of an unsuitable frame is when a large number of pixels exhibit vertical disparity, i.e., when the representation of a particular object in one image of a stereoscopic pair is vertically offset compared to the representation of that object in the other image of the stereoscopic pair. Unlike horizontal disparity offset, vertical disparity offset does not contribute to the illusion of depth perceived in a stereoscopic pair and may indicate insufficient 3D effect or insufficient disparity estimation. Similarly, a frame that contains a high percentage of pixels with horizontal disparity below a predetermined threshold (e.g., 5 pixels) may indicate that the applied 3D effect is minimal based on the disparity value estimated for that frame.

[0030] In some embodiments where the training engine 122 trains the machine learning model 230 to perform stereoscopic transformation on images, the training data pre-processing engine 220 may select representative frames from a shot after filtering out irrelevant frames, as previously described. In other embodiments where the training engine 122 trains the machine learning model 230 to perform stereoscopic transformation on image sequences, the training data pre-processing engine 220 may select representative frame sequences from a particular shot after further filtering out frame sequences of shots that exhibit little or no object motion (i.e., still shots). For example, the training data pre-processing engine 220 may set a minimum object motion threshold between adjacent frames. In one non-limiting example, the minimum object motion threshold may be 8 pixels between adjacent frames.

[0031] The machine learning model 230 includes a disparity estimation network 235. The disparity estimation network 235 receives as input pre-processed training data, which is either a CGI image or a frame of a representative frame sequence from an existing 3D movie sequence. In various embodiments where the input image to the disparity estimation network 235 is a frame of a representative frame sequence from an existing 3D movie sequence, the disparity estimation network 235 creates a reduced resolution representation of one image from that frame (e.g., the left image or the right image) and predicts the depth d of each object and background in that image. The depth d represents the distance between the camera and the object or background shown in the image. Since images often depict background objects such as the sun or sky at a distant or near infinite distance, in various embodiments the disparity estimation network 235 predicts the inverse of the depth (1 / d). Very large depth values ​​associated with distant objects can complicate the estimation calculation and introduce errors. By calculating the reciprocal depth for an object, the reciprocal depth for very distant objects tends to approach zero, thus avoiding very large numbers in the next estimation calculation. The resulting reciprocal depth estimates are then recorded pixel by pixel using a reduced resolution image.

[0032] The disparity estimation network 235 aligns the per-pixel inverse depth information for the reduced resolution images to produce disparity, i.e., per-pixel displacement vectors. This alignment step generates for each pixel a horizontal displacement vector proportional to the inverse of the pixel's estimated depth. In this way, pixels representing foreground (nearby) objects receive larger displacement vectors and thus represent larger disparity values ​​than pixels representing distant objects. These different horizontal displacements create the 3D effect of perceived depth in the output image predicted by the machine learning model 230.

[0033] In various embodiments where the input provided to the disparity estimation network 235 is a CGI image associated with a ground truth disparity value, the disparity estimation network 235 aligns the estimated disparity value with the ground truth disparity value using least squares alignment. In various other embodiments where the input to the disparity estimation network 235 is a representative sequence of frames from an existing 3D movie sequence, the disparity estimation network 235 calculates, for each frame of the representative sequence of frames, a ground truth disparity value from the corresponding left and right images, and aligns the estimated disparity value with the calculated ground truth disparity value using least squares alignment.

[0034] In some embodiments, the disparity estimation network 235 performs disparity estimation on an optional reduced resolution version of the input. The input can be a CGI image or a frame of a representative frame sequence from an existing 3D movie sequence. If the input is a frame of a representative frame sequence from an existing 3D movie sequence, the disparity estimation network 235 performs disparity estimation on either the right or left image of the frame. To generate a high-resolution disparity estimate, the disparity estimation network 235 upsamples from the low-resolution disparity estimate and the high-resolution input. Specifically, the disparity estimation network 235 uses bilateral upsampling with a neighborhood attention mechanism. In bilateral upsampling, the low-resolution disparity estimate is resampled at a higher resolution to match the input using a bilateral filter that preserves object edges in the high-resolution disparity estimate and smooths flat regions in the high-resolution disparity estimate. The neighborhood attention mechanism prioritizes disparity values ​​from nearby pixels when generating disparity values ​​for newly generated pixels in the high-resolution disparity estimate.

[0035] As shown in Fig. 2, the machine learning model 230 also includes a transformation network 240. The disparity values ​​estimated by the disparity estimation network 235 are training inputs to the transformation network 240 of the machine learning model 230 and serve as trainable weight parameters. To avoid excessively large weight values ​​in the transformation network 240, the disparity estimation network 235 of various embodiments normalizes the estimated disparity values. Minimum and maximum desired disparity values ​​are predetermined for the disparity estimation network 235, and the disparity estimation network 235 normalizes the generated disparity values ​​to this range by applying a linear operator.

[0036] In operation, the transformation network 240 performs feature extraction on the input image, warps the feature space representation of the input image by the image's estimated disparity value, performs splatting to assign multiple pixels of the input image to the same pixels in the warped image, and interpolates image regions where previously occluded background regions are exposed by the warping. The transformation network 240 combines the warped, splatted, and interpolated feature space representation of the input image into a predicted output image 250.

[0037] The input to the transformation network 240 is an input image and a high-resolution disparity estimate associated with that image. In various embodiments, the input to the transformation network 240 may be a left image, for which it generates a corresponding predicted right image, or may be a right image, for which it generates a corresponding predicted left image. In various other embodiments, the input may be a center image, for which it generates both left and right output images (e.g., sequentially). In various embodiments where the right image is generated from an input left image, or where the left image is generated from an input right image, the input image and the generated predicted output images form a stereo pair. In various other embodiments where it generates both a left image and a right image from an input center image, the generated left and right images form a stereo pair.

[0038] The transformation network 240 includes a feature extraction network 242. The feature extraction network 242 performs pyramidal feature extraction on the input image at various levels of resolution. For each level of resolution, the feature extraction network 242 generates a feature vector that represents the input image at that resolution. The feature vector generated for a particular resolution will be different from the feature vector generated for a different resolution. The multiple feature vectors collectively form a multi-resolution feature vector that represents the input image in a feature space, rather than a native representation of the input image as a matrix of RGB pixels.

[0039] The transformation network 240 further includes a warping and splatting network 244 that warps the feature space representation of the input image by the per-pixel displacement vectors calculated by the disparity estimation network 235. Specifically, each pixel of the feature space representation of the input image is shifted by a displacement vector proportional to the disparity value estimated for that pixel in the high-resolution disparity estimation.

[0040] The warped feature space representation of the input image contains disoccluded regions, where previously hidden areas of the background are exposed due to the displacement of objects in the scene. These disoccluded regions are left unfilled by the warping process and are filled in during a subsequent inpainting step. The warped representation also contains occluded regions, where multiple pixels from the input image are displaced to the same location in the feature space representation. The warping and splatting network 244 tackles these occluded regions using a differentiable warping strategy that includes weights that determine the order of multiple pixels warped to the same location. This order determines which of the multiple pixels displaced (i.e., splatted) to the same location in feature space should remain visible. The weights are generated from the per-pixel disparity values ​​estimated by the disparity estimation network 235 as described above. Since foreground objects have smaller estimated depth values ​​compared to background objects, the pixels of the foreground objects are given larger disparity values ​​by the disparity estimation network 235. Thus, the disparity value directly indicates which splatted pixels are foreground and should remain visible, and which are background and should remain hidden. Before using the disparity values ​​to determine the order of the splatted pixels, the warping and splatting network 244 normalizes the disparity values ​​to the range [-1,1] and scales the normalized values ​​with a trainable parameter α.

[0041] The warping and splatting network 244 performs the warping and splatting process using the multi-resolution feature vectors of all levels. For each resolution level, the warping and splatting network 244 upsamples the estimated disparity values ​​by interpolation to match the particular resolution level.

[0042] The warping and splatting network 244 also warps the original input image at the original resolution and at scaled resolutions corresponding to all levels of the extracted feature pyramid, thereby providing a user-visible representation of the warping and splatting effect. The warping and splatting network 244 also generates a warping mask that specifies how many pixels are splatted at each location in the warped input image. The warping and splatting network 244 also generates a binary disocclusion mask that specifies which pixels in the warped input image represent disoccluded background regions that require inpainting.

[0043] The transformation network 240 includes an image synthesis network 246 that performs background-aware inpainting of disoccluded regions and addresses sharp edges and aliasing in occluded regions with synthesis refinement. Finally, the image synthesis network 246 generates a predicted output image 250.

[0044] Given the warped image features and disparities for multiple resolutions as input, the image synthesis network 246 performs background-aware inpainting and synthetic refinement. Inpainting refers to the completion of a background region that was previously hidden but is now disoccluded by one or more foreground objects. Synthetic refinement reduces aliasing and sharp edges at the new boundary between the foreground object and the background where the previously exposed background is now covered and occluded by one or more foreground objects. The image synthesis network 246 performs background-aware inpainting by applying a neighborhood-based self-attention technique to all resolution layers with local multi-scale self-attention. In various embodiments, the image synthesis network 246 includes a single-headed attention layer with a predefined neighborhood size. This neighborhood-based self-attention technique ensures that disoccluded regions are inpainted with content based only on local background regions. After warping, background-aware inpainting, and synthetic refinement, the image synthesis network 246 converts the multi-resolution feature space representation into a final expected output image. In various embodiments, the image synthesis network 246 performs the transformation using a GridNet convolutional neural network architecture that includes 3 rows and 6 columns, with bilinear upsampling replacing the transposed convolution in the network. In various embodiments, all steps of the stereoscopic transformation are differentiable. As a result, the various networks can be trained together end-to-end.

[0045] When training the machine learning model 230, the disparity estimation network 235 is first trained separately. Alternatively, the training engine 122 may use a pre-trained version of the disparity estimation network 235.

[0046] The feature extraction network 242 and the image synthesis network 246 are trained by training the entire machine learning model 230 end-to-end while holding the weights of the disparity estimation network 235 constant. The training is supervised, using either CGI images with ground truth disparity information or left and right image pairs of a stereoscopic video dataset with associated and annotated disparity information. The supervised training optionally includes data augmentation, where the image pairs of the stereoscopic video dataset are flipped either horizontally or vertically (e.g., with a probability of 0.5 / 0.1, respectively) to generate additional left and right image pairs for training. If the left and right image pairs of the stereoscopic video dataset contain only negative disparity information, an offset may be applied to the right image of the pair to obtain both negative and positive disparity. The offset is determined by the median of the ground truth image disparity plus a random number between -10 and 10.

[0047] The disparity estimation network 235 generates predicted disparities, which are then aligned to the left and right disparities of the dataset using a least squares method. In some embodiments, training continues for 200,000 steps using a batch size of 4 and a learning rate of 1e-3.

[0048] In the final step, the training engine 122 trains the entire pipeline end-to-end without fixing the weights of the disparity estimation network 235. The training is not only supervised using left and right image pairs, but also uses the disparity information of the dataset as an intermediate teacher signal for the disparity estimation network 235. This intermediate teacher signal is used as a weighting term in the loss function and can be faded out progressively during training.

[0049] During training, the training engine 122 compares the frame loss 280 with the expected output image 250 TIFF2024169343000002.tif6114

[0050] and the ground truth right frame I rFor the pixel-by-pixel comparison of, we compute a separate disocclusion loss 260 weight β for the disocclusion region D as the L1 loss function (Equation 1).

[0051]

number

[0052] Additionally, the training engine 122 calculates the disparity and perceptual loss 270. F represents the difference between the expected output image of high-level semantic features and the ground truth right frame of the training data. d denotes the L1 distance between the expected disparity values ​​from the disparity estimation network 235 and the known training dataset disparities. Thus, the total loss function is

[0053]

number

[0054] It is.

[0055] FIG. 3A is a diagram illustrating the stereoscopic transformation engine 124 of FIG. 1 in more detail, according to some embodiments. In these embodiments, the stereoscopic transformation engine 124 modifies a single input image 300 (e.g., a still image or a single frame from a video sequence) to generate an expected output image 345. The input image 300 and the generated expected output image 345 form a stereoscopic pair of images. When viewed together, the stereoscopic pair depicts a scene that provides the illusion of relative depth between various objects in the scene and between the objects and the background. In various embodiments, the input image may be a left image, for which a corresponding expected right image is generated, or may be a right image, for which a corresponding expected left image is generated. In various other embodiments, the input is a center image, for which both a left and a right output image are generated (e.g., sequentially).

[0056] The stereoscopic transformation engine 124 includes a depth predictor 305, which modifies the input image 300 to produce a reduced resolution version of the input image. The depth predictor 305 also predicts the depth d of objects and background in the reduced resolution version of the input image. The depth d represents the distance between the camera and the object or background shown in the reduced resolution version of the input image. Since the input images often depict background objects, such as the sun or sky, at a distant or near infinite distance, in various embodiments, the depth predictor 305 predicts the inverse of the depth (1 / d). The resulting inverse depth estimate is recorded for each pixel using the reduced resolution version of the input image.

[0057] The stereoscopic transformation engine 124 performs disparity estimation 310 on the reduced resolution version of the input image and the reciprocal depth estimates associated with the reduced resolution version of the input image. The disparity estimation 310 aligns the per-pixel reciprocal depth information for the reduced resolution version of the input image to produce a disparity, i.e., a per-pixel displacement vector. This alignment step generates a horizontal displacement vector for each pixel that is proportional to the inverse of the pixel's estimated depth. In this way, pixels representing foreground (nearby) objects receive a larger horizontal displacement vector, representing a larger disparity value, compared to pixels representing distant objects. These different horizontal displacements create a 3D effect of perceived depth in the expected output image 345 when viewed together with the input image 300. To avoid excessively large values ​​in the stereoscopic transformation engine 124, various embodiments of the disparity estimation 310 normalize the estimated disparity values. The minimum and maximum desired disparity values ​​are predetermined, and the disparity estimation 310 normalizes the generated disparity values ​​to this range by applying a linear operator. In various other embodiments, the user provides the disparity estimation 310 with a reference disparity in the form of a manual annotation of one or more portions of the input image 300 selected by the user (e.g., using paint strokes, defining a bounding box, or manually selecting an object in the image). The disparity estimation 310 aligns the expected disparity to the given reference disparity value, for example, using a least-squares alignment technique.

[0058] Disparity upsampling 315 modifies a reduced resolution version of the input image and the associated disparity estimate to generate a high-resolution predicted disparity map 320. Disparity upsampling 315 upsamples from a low-resolution disparity estimate and a high-resolution input image. Specifically, disparity upsampling 315 uses bilateral upsampling with a neighborhood attention mechanism. In bilateral upsampling, the low-resolution disparity estimate is resampled at a higher resolution to match the input image using a bilateral filter that preserves the edges of objects in the high-resolution predicted disparity map 320 and smooths flat regions in the high-resolution predicted disparity map 320. The neighborhood attention mechanism prioritizes disparity values ​​from nearby pixels when generating disparity values ​​for newly generated pixels in the high-resolution predicted disparity map 320.

[0059] The feature extraction 322 performs pyramidal feature extraction on the input image at various levels of resolution. For each level of resolution, the feature extraction 322 generates a feature vector that represents the input image at that resolution. The feature vector generated for a particular resolution will be different from the feature vector generated for a different resolution. The multiple feature vectors collectively form a multi-resolution feature vector that represents the input image in a feature space, rather than a native representation of the input image as a matrix of RGB pixels.

[0060] Depth-aware splatting 325 warps the feature space representation of the input image by the per-pixel displacement vectors in the expected disparity map 320. Specifically, each pixel of the feature space representation of the input image is shifted by a displacement vector proportional to the disparity value estimated for that pixel in the expected disparity map 320.

[0061] The feature space representation of the warped input image contains disoccluded regions, where previously hidden areas of the background are exposed due to the displacement of objects in the scene. These disoccluded regions remain unfilled by the warping process and are filled in during a subsequent inpainting step. The warped representation also contains occluded regions, where multiple pixels from the input image are displaced to the same location in the feature space representation. Depth-Aware Splatting 325 tackles these occluded regions using a differentiable warping strategy that includes weights that determine the order of multiple pixels warped to the same location. This order determines which of the multiple pixels displaced (i.e., splatted) to the same location in feature space should remain visible. The weights are generated from the per-pixel disparity values ​​of the expected disparity map 320. Foreground objects will have smaller estimated depth values ​​compared to background objects, so the pixels of the foreground objects are given larger disparity values ​​by the disparity estimation 310. Thus, the disparity value directly indicates which splatted pixels are foreground and should remain visible, and which are background and should remain hidden. Before using the disparity values ​​to determine the order of the splatted pixels, depth-aware splatting 325 normalizes the disparity values ​​to the range [-1,1] and scales the normalized values ​​with a parameter α determined during training.

[0062] Depth-aware splatting 325 applies the above warping and splatting process with the multi-resolution feature vectors of all levels. For each resolution level, depth-aware splatting 325 upsamples the estimated disparity values ​​by interpolation to match the particular resolution level.

[0063] Depth-Aware Splatting 325 also warps the original input image at the original resolution and at scaled resolutions corresponding to all levels of the extracted feature pyramid, thereby providing a user-visible representation of the warping and splatting effects. Depth-Aware Splatting 325 also generates a warping mask that specifies how many pixels are splatted at each location in the warped input image. Depth-Aware Splatting 325 also generates a binary disocclusion mask that specifies which pixels in the warped input image represent disoccluded background regions that require inpainting.

[0064] Given the warped image features and disparities for multiple resolutions, image synthesis 330 performs inpainting 335 and synthetic refinement 340. Inpainting refers to the completion of newly disoccluded background regions that were previously hidden by one or more foreground objects. Synthetic refinement reduces aliasing and sharp edges at new boundaries between foreground objects and background in regions where previously exposed background is now covered and occluded by one or more foreground objects. Inpainting 335 performs background-aware inpainting by applying a neighborhood-based self-attention technique to all resolution layers using local multi-scale self-attention. In various embodiments, inpainting 335 includes a single-headed attention layer with a predefined neighborhood size. This neighborhood-based self-attention technique ensures that disoccluded regions are inpainted with content based only on local background regions.

[0065] After warping, background-aware inpainting, and synthesis refinement, Image Synthesis 330 converts the multi-resolution feature space representation into a final expected output image 345. In various embodiments, Image Synthesis 330 performs the conversion using a GridNet convolutional neural network architecture that includes 3 rows and 6 columns, with bilinear upsampling replacing transposed convolutions in the network.

[0066] 3B illustrates the stereoscopic transformation engine 124 of FIG. 1 in more detail, according to some embodiments. In these embodiments, the stereoscopic transformation engine 124 modifies an input image 350 (e.g., a single 2D video frame) and an input keyframe 352 to generate an expected output image 395. The input image 350 and the generated expected output image 395 form a stereoscopic pair of images. In various embodiments, the input image 350 may be a left image for which a corresponding expected right image is generated, or may be a right image for which a corresponding expected left image is generated. In various other embodiments, the input is a center image for which both a left and a right output image are generated (e.g., sequentially).

[0067] The input keyframe 352 is a video frame that is temporally related to the input image 350 (i.e., the input keyframe 352 precedes or follows the input image 350 in the video frame sequence). In some embodiments, the input keyframe 352 is selected from a fixed set of neighboring video frames TIFF2024169343000005.tif8114

[0068] However, TIFF2024169343000006.tif7114

[0069] represents the input image 350. In other embodiments, the input keyframes 352 are frames at exponentially increasing distances from the input image 350, e.g. TIFF2024169343000007.tif8114

[0070] The input keyframes 352 provide a temporal context for the input image 350 to improve the accuracy and stability of the predicted output image 395.

[0071] The stereoscopic transformation engine 124 performs a stereoscopic transformation on the video sequence by processing multiple sequential input images 350 along with selected keyframes 352 associated with each input image 350 .

[0072] The stereoscopic transformation engine 124 includes a depth predictor 355. The depth predictor 305 modifies the input image 350 to produce a reduced resolution version of the input image. The depth predictor 355 further predicts the depth d of objects and the background in the reduced resolution version of the input image. In various embodiments, the depth predictor 355 predicts the inverse depth (1 / d), as already described with respect to FIG. 3A. The resulting inverse depth estimate is recorded for each pixel using the reduced resolution version of the input image.

[0073] For any given frame (i.e., input image 350 and input keyframe 352), depth prediction 355 further generates a scene point cloud by analyzing changing or stationary object and background positions over multiple frames. Depth prediction 355 further estimates the camera pose and position. To estimate the camera pose and position, depth prediction 355 may use, for example, structure-from-motion techniques. For each time step, depth prediction 355 generates a sparse depth rendering of the point cloud represented by input image 350 and input keyframe 352. Depth prediction 355 generates a reduced resolution representation of input image 350 and each keyframe 352 that includes the inverse depth value for each pixel.

[0074] The stereoscopic transformation engine 124 performs disparity estimation 360 on the reduced resolution version of the input image and the reciprocal depth estimate associated with the reduced resolution version of the input image. Similar to disparity estimation 310 above, disparity estimation 360 aligns the per-pixel reciprocal depth information for the reduced resolution version of the input image to produce a normalized expected disparity value for each pixel. In various other embodiments, a user provides a reference disparity to disparity estimation 360 in the form of a manual annotation of one or more portions of the input image 350 selected by the user. Disparity estimation 360 aligns the expected disparity to the provided reference disparity value using, for example, a least squares alignment technique.

[0075] As previously described for disparity upsampling 315, disparity upsampling 365 modifies a reduced resolution version of the input image and the associated disparity estimate to generate a high resolution predicted disparity map 370. Disparity upsampling 365 upsamples from the low resolution disparity estimate and the high resolution input image to generate a high resolution predicted disparity map 370.

[0076] The stereoscopic transformation engine 124 generates an expected output image 395 from the input image 350. For example, the input image 350 may be TIFF2024169343000008.tif6114

[0077] , the expected output image 395 may be the corresponding right image TIFF2024169343000009.tif5114

[0078] where the input image 350 and the expected output image 395 may form a stereo pair. The stereo transformation engine 124 may also consider additional images (i.e., keyframes) from other time instances. In the following description, the stereo transformation engine 124 may consider a single additional image TIFF2024169343000010.tif6114

[0079] only considers 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,

[0080] The stereoscopic transformation engine 124 generates motion vectors in the motion vector generation 372 to detect the motion of the key frame (in this case, TIFF2024169343000011.tif7114

[0081] ) to predict the output image 395 (in this case, TIFF2024169343000012.tif6114

[0082] In some embodiments, the motion vector generation 372 warps the forward vector to the key frame TIFF2024169343000013.tif7114

[0083] and the input image TIFF2024169343000014.tif6114

[0084] The motion vector generator 372 generates a disparity map based on the forward optical flow between time instance t and time instance t, and the disparity value for time instance t from the predicted disparity map 370. The forward optical flow includes a pixel-by-pixel vector that represents the motion of objects depicted in the scene between time instance t-1 and time instance t. The motion vector generator 372 then reverse warps the disparity value for time instance t to time instance t-1. The motion vector generator 372 then adds the warped disparity value to the optical flow vector to obtain TIFF2024169343000015.tif7114

[0085] from TIFF2024169343000016.tif6114

[0086] Resulting in a forward vector to

[0087] In another embodiment, the motion vector generation 372 generates a backward vector based on the input frame TIFF2024169343000017.tif6114

[0088] and keyframes TIFF2024169343000018.tif7114

[0089] The motion vector generation 372 then generates a motion vector based on the backward optical flow between time instances t and t−1. The backward optical flow includes pixel-by-pixel vectors that represent the motion of objects depicted in the scene between time instances t and t−1. TIFF2024169343000019.tif6114

[0090] The motion vector generation 372 also generates reverse (right-to-left) disparity values ​​by running the stereoscopic transformation engine 124 on the input image as already described for Figure 3A to calculate the negative of the warped per-pixel disparity values ​​obtained after inpainting 335 of Figure 3A. The motion vector generation 372 then adds the warped right-to-left disparity values ​​to the reverse optical flow to produce a reverse vector.

[0091] In yet another embodiment, the motion vector generation 372 generates predicted vectors directly based on an estimate of the output image generated from the input image as already described for Figure 3 A. Given the estimated output image, the motion vector generation 372 directly estimates the optical flow between the input image and the estimated output image and calculates the forward or backward vectors.

[0092] Feature Extraction and Depth-Aware Splatting 375 performs pyramidal feature extraction on the input image 350 and each keyframe 352 at various levels of resolution. For each level of resolution, Feature Extraction and Depth-Aware Splatting 375 generates a feature vector that represents the input image or keyframe at that resolution. Collectively, multiple feature vectors form a multi-resolution feature vector that represents the input image or keyframe in feature space.

[0093] Feature Extraction and Depth Considerate Splatting 375 warps the feature space representation of the input image by the motion vectors generated by Motion Vector Generation 372. Specifically, each pixel in the feature space representation of the input image is shifted by the value of the generated motion vector. In various embodiments that include multiple keyframes 352, motion vectors are generated as described above between each keyframe 352 and the input image 350. Feature Extraction and Depth Considerate Splatting 375 then warps the feature space representation of the input image based on each of these multiple motion vectors.

[0094] The warped feature space representation of the input image includes disoccluded regions that remain uncompleted by the warping process and are completed during a subsequent inpainting step. The warped feature space representation also includes occluded regions, where multiple pixels from the input image are displaced to the same location in the feature space representation. Feature Extraction and Depth-Aware Splatting 375 addresses these occluded regions using a differentiable warping strategy that includes weights that determine the order of multiple pixels warped to the same location. The weights are generated from the per-pixel disparity values ​​of the expected disparity map 370 in the same manner as described for depth-aware splatting 325 already. This weighted order determines which of multiple pixels displaced (i.e., splatted) to the same location in feature space should remain visible.

[0095] Feature Extraction and Depth-Aware Splatting 375 performs the above warping and splatting process using the multi-resolution feature vectors of all levels. For each resolution level, Feature Extraction and Depth-Aware Splatting 375 upsamples the estimated disparity values ​​by interpolation to match the particular resolution level.

[0096] Feature Extraction and Depth-Aware Splatting 375 also warps the original input image at the original resolution and at scaled resolutions corresponding to all levels of the extracted feature pyramid, thereby providing a user-visible representation of the depth-aware warping and splatting effect. Feature Extraction and Depth-Aware Splatting 375 also generates a warping mask that specifies how many pixels are splatted at each location in the warped input image. Feature Extraction and Depth-Aware Splatting 375 also generates a binary disocclusion mask that specifies which pixels in the warped input image represent disoccluded background regions that require inpainting.

[0097] Using the warped image features and disparities for multiple resolutions, image synthesis 380 performs inpainting 385 and synthetic refinement 390. Inpainting and synthetic refinement complete previously hidden, newly disoccluded background regions and reduce aliasing and sharp edges at new boundaries between foreground objects and background in occluded regions. Inpainting 385 performs background-aware inpainting by applying neighborhood-based self-attention techniques using local multi-scale self-attention as described above for inpainting 335. In addition, inpainting 385 may analyze input images 350 and keyframes 352 to find background regions that were previously visible in one or more keyframes 352 but are occluded in input image 350, and refine the inpaint using these previously visible regions. For example, a particular section of the background may be visible in keyframe 352 representing time instance t-1, but is occluded in input image 350 representing time instance t. Inpaint 385 may identify that particular section of the background as newly disoccluded in the expected output image 395 and use the background information from the keyframe representing time instance t-1 to more accurately inpaint the newly disoccluded area.

[0098] After warping, background-aware inpainting, and synthesis refinement, Image Synthesis 380 converts the multi-resolution feature space representations of input images 350 and keyframes 352 into a final expected output image 395. In various embodiments, Image Synthesis 380 performs the conversion using a GridNet convolutional neural network architecture that includes 3 rows and 6 columns, with bilinear upsampling replacing transposed convolutions in the network.

[0099] 4 shows an example input image and the results of various warping and disparity estimation applied to the input image according to some embodiments. The top left panel of FIG. 4 shows an input image 400. In various embodiments, the input image 400 is, for example, a still image or a single frame of a 2D video sequence. The input image 400 depicts an object in the foreground of the image and also depicts a background that is partially occluded by the object.

[0100] The top right panel of FIG. 4 shows the results of depth-aware splatting 325, as already described for FIG. 3A. Depth-aware splatting 325 offsets each pixel of the feature space representation of the input image 400 by an amount proportional to the disparity value estimated for that pixel. The pixel offsets produced by depth-aware splatting 325 result in occlusions 404 and disocclusions 402. Depth-aware splatting 325 correctly places foreground pixels in front of newly occluded background pixels, ensuring that boundary edges between foreground objects and the background are not distorted. The boundaries may contain sharp edges and aliasing artifacts, which are addressed in the following synthesis refinement 340, as already described for FIG. 3A. Disocclusion 402 correctly paints pixels representing newly exposed areas of the background. Inpainting 335, as already described, completes the disoccluded areas using only background pixels.

[0101] The bottom left panel of FIG. 4 shows a disparity representation 406 of the expected disparity map 320. As already described with respect to FIG. 3A, each pixel of the disparity map 320 includes an estimated disparity value. The depth-aware splatting 325 warps each pixel of the feature space representation of the input image with an offset proportional to the disparity value associated with that pixel. In the disparity representation 406, lighter regions represent foreground pixels closer to the observer and have larger estimated disparity values, while darker regions represent background pixels further from the observer and have smaller estimated disparity values. In various embodiments, the disparity values ​​in the disparity representation 406 may be drawn using any suitable annotation scheme, for example, darker pixels represent larger estimated disparity values ​​and lighter pixels represent smaller estimated disparity values. The disparity representation 406 may also use different colors to represent different ranges of disparity values.

[0102] The center bottom panel of Figure 4 shows the warping mask 408. As already described for Figure 3A, depth-aware splatting 325 generates the warping mask 408. The warping mask 408 represents the newly occluded regions in the warped input image 400 and also represents how many pixels are splatted into each location of the warped input image.

[0103] The bottom right panel of Figure 4 shows a binary disocclusion mask 410. Depth-aware splatting 325 generates a binary disocclusion mask 410 that records which pixels of the warped input image represent newly disoccluded background regions that are to be filled in with background pixels as already described for inpainting 335.

[0104] 5 is a flow chart of method steps for training a deep learning model to perform stereoscopic transformations according to various embodiments. The method steps are described with respect to the systems of FIGS. 1-3, but one of ordinary skill in the art will appreciate that any system configured to perform the method steps in any order is within the scope of the present disclosure.

[0105] As shown, in operation 502, the training engine 122 executes a first neural network to generate inverse depth values ​​associated with objects in an input image. In various embodiments, the input image is a still image or a single frame from a video sequence. The training engine 122 generates inverse depth values ​​associated with each pixel of the input image. The training engine 122 also generates disparity values ​​associated with each pixel of the input image.

[0106] In operation 504, the training engine 122 trains the parameters of the first neural network using backpropagation and a disparity loss L d The disparity loss L d denotes the L1 distance between the expected disparity values ​​from the first neural network and the known training dataset disparity values.

[0107] In operation 506, the training engine 122 compares the disparity loss from the first neural network to a first predetermined threshold. If the first predetermined threshold is not met, then the training engine 122 returns to operation 502 and continues training the first neural network until the first predetermined threshold is met.

[0108] If the predetermined threshold is met, then the training engine 122 proceeds to operation 508. In operation 508, while holding the trained weights of the first neural network constant, the training engine 122 pre-trains a second set of neural networks. The training engine 122 performs supervised training on the second set of neural networks using a dataset of left and right image pairs and disparity values ​​associated with the image pairs. As described with respect to Figures 3A and 3B, the second set of neural networks performs feature extraction using the input images and disparity values ​​associated with the input images. The second set of neural networks further performs depth-aware splatting, background-aware inpainting, and synthetic refinement to generate expected output images.

[0109] In operation 510, the training engine 122 updates the parameters of the second set of neural networks using backpropagation. As previously described with respect to FIG. 2, the training engine 122 calculates the frame loss as an L1 loss function L for the pixel-by-pixel comparison of the expected output image and the ground truth right frame of the training data for the image pair. L1 In addition, the training engine 122 calculates the disparity and perceptual loss. F represents the difference between the expected output image of the high-level semantic features and the ground truth right frame of the training data. d denotes the L1 distance between the predicted disparity values ​​from the first neural network and the known training dataset disparities. The perceptual loss L F and the disparity loss L d Each of the vectors has trainable weight parameters (γ and δ, respectively). The total loss function is then given by equation (2), as already described for FIG.

[0110]

number

[0111] In operation 512, the training engine 122 compares the total loss function given by equation (2) to a second predetermined threshold. If the second predetermined threshold is not met, the training engine 122 continues with the second set of neural networks until the second predetermined threshold is met.

[0112] In operation 514, the training engine 122 runs the first neural network and the second set of neural networks end-to-end. Unlike operation 508 above, the parameters of the first neural network are no longer fixed.

[0113] Next, in operation 516, the training engine 122 performs supervised training using the dataset of left and right image pairs together with ground truth disparity values ​​associated with the dataset image pairs. In some embodiments, these ground truth disparity values ​​may be used as intermediate teacher signals to train the first neural network. The intermediate teacher signals may be progressively faded out during training. All functions and transformations in the first neural network and the second set of neural networks are fully differentiable. Thus, the training engine 122 trains both the first neural network and the second set of neural networks end-to-end using backpropagation. The training engine 122 generates frame, perceptual, and disparity losses, and also generates a total loss function. These loss functions are the same as those already described for operation 510.

[0114] In operation 518, training engine 122 compares the total loss function determined in operation 516 to a third predetermined threshold. If the third predetermined threshold is not met, training engine 122 continues training the first neural network and the second set of neural networks until the third predetermined threshold is met, terminating end-to-end training.

[0115] 6 is a flow chart of method steps for performing a stereoscopic transformation on a single input image according to various embodiments. The method steps are described with respect to the systems of Figures 1-3, but one of ordinary skill in the art will appreciate that any system configured to perform the method steps in any order is within the scope of the present disclosure.

[0116] As shown, in operation 602, the stereoscopic transformation engine 124 accepts an input image. In various embodiments, the input image may be a still image or a single frame from a video sequence. In various embodiments, the stereoscopic transformation engine also accepts additional user input representing reference disparity information in the form of manual annotation of one or more portions of the input image selected by a user (e.g., using paint strokes, defining a bounding box, or manually selecting an object in the image).

[0117] In operation 604, the stereoscopic transformation engine 124 generates a predicted disparity map for the input image. The stereoscopic transformation engine 124 transforms the input image into a low-resolution representation of the input image and executes a first trained neural network to perform depth prediction of various objects and background depicted in the low-resolution representation. The stereoscopic transformation engine 124 then calculates and stores the inverse depth value associated with each pixel of the low-resolution representation of the input image. For each pixel, the stereoscopic transformation engine 124 estimates a disparity value proportional to the inverse depth value associated with the pixel. The estimated disparity values ​​are aligned to further user input, for example, by a least squares alignment method, if available. These estimated disparity values ​​represent the horizontal offset that needs to be applied for each pixel in the next warping and splatting operation. The stereoscopic transformation engine 124 then upsamples the disparity values ​​associated with the low-resolution representation of the input image to the full resolution of the input image, as already described with respect to FIG. 3A. The stereoscopic transformation engine 124 converts the upsampled disparity estimate into a full-resolution expected disparity map for the input image, where each pixel in the disparity map has an associated disparity value, and pixels representing foreground objects have larger associated disparity values, while background objects have smaller associated disparity values.

[0118] In operation 606, the stereoscopic transformation engine 124 performs pyramidal feature extraction on the input image at different levels of resolution. For each level of resolution, the stereoscopic transformation engine 124 generates a vector representing the features of the input image at the selected resolution. The feature vectors generated for the different resolutions will be distinct from each other. The multiple feature vectors collectively form a multi-resolution feature vector that represents the input image in a feature space, rather than a native representation of the input image as a matrix of RGB pixels.

[0119] Next, in operation 608, the stereo transformation engine 124 modifies the feature space representation for the input image by performing warping and depth-aware splatting on the extracted feature data using a second set of neural networks. Each pixel in the feature space representation is warped (offset) by an amount proportional to the disparity value associated with that pixel. Because pixels at various depths have different disparity values, multiple pixels may be warped to the same location in the modified feature space representation, a process described as splatting. Splatting occurs at boundaries between foreground and background objects where pixels in both the foreground and background are warped to the same location. The stereo transformation engine 124 determines which splatted pixels should remain visible in the modified feature space representation based on the disparity value associated with each warped pixel. Pixels with relatively large associated disparity values ​​represent objects closer to the viewer and should remain visible. Splatted pixels with relatively small disparity values ​​represent background objects that are far from the observer and should remain hidden in the modified feature space representation.

[0120] In operation 610, the stereo transformation engine 124 further modifies the feature space representation by inpainting the disoccluded regions with respect to their background. Disoccluded regions are regions of the input image that were previously occluded by one or more foreground objects but that have become visible due to the warping process. The stereo transformation engine 124 artificially inpaints these regions using only information from the image background, as already described with respect to FIG. 3A. The stereo transformation engine 124 also synthetically refines the splatted pixels to remove sharp edges and aliasing artifacts.

[0121] In operation 612, the stereoscopic transformation engine 124 transforms the warped, splatted, and inpainted feature space representation into a predicted output image that, when viewed together with the input image, forms a stereoscopic pair that provides the illusion of different relative depths for the various objects depicted in the images.

[0122] 7 is a flow chart of method steps for performing stereoscopic transformation on a single video frame as an input image using neighboring video frames as input keyframes according to various embodiments. The method steps are described with respect to the systems of FIGS. 1-3, however, one skilled in the art will appreciate that any system configured to perform the method steps in any order is within the scope of the present disclosure.

[0123] As shown, in operation 702, the stereoscopic transformation engine 124 accepts a frame from a video sequence as an input image, and also accepts one or more neighboring video frames as input keyframes. In various embodiments, the input image may be a left image, for which it generates a corresponding predicted right image, or may be a right image, for which it generates a corresponding predicted left image. In various other embodiments, the input is a center image, for which it generates both left and right output images (e.g., sequentially).

[0124] An input keyframe is a video frame that is temporally related to the input image (i.e., the input keyframe precedes or follows the input image 350 in the video frame sequence). In some embodiments, the input keyframe is selected from a fixed set of neighboring video frames TIFF2024169343000021.tif8114

[0125] However, TIFF2024169343000022.tif7114

[0126] represents the input image. In other embodiments, the input keyframes are frames at exponentially increasing distances from the input image, e.g. TIFF2024169343000023.tif7114

[0127] The input keyframes provide a temporal context for the input images to improve the accuracy and stability of the expected output images. The stereoscopic transformation engine 124 performs a stereoscopic transformation on a video sequence by processing multiple sequential input images, along with selected keyframes associated with each input image. In the following description, the stereoscopic transformation engine 124 performs a stereoscopic transformation on a video sequence by processing multiple sequential input images, along with selected keyframes associated with each input image. TIFF2024169343000024.tif6114

[0128] only considers 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,

[0129] In operation 704, the stereo transformation engine 124 generates a high-resolution predicted disparity map for the input image and for each input key frame. The stereo transformation engine 124 performs depth prediction by first generating a reduced resolution version of the input image and each input key frame using a first neural network. The stereo transformation engine 124 then predicts the depth of various objects and background of the image. In various embodiments, the stereo transformation engine 124 predicts the inverse of the depth 1 / d, as already described with respect to FIG. 3B. The stereo transformation engine 124 further generates a scene point cloud by analyzing the positions of changing or stationary objects and background across the input frames and one or more input key frames. The stereo transformation engine 124 estimates the camera pose and configuration, for example, using structure-from-motion techniques. For each time step, the stereo transformation engine 124 generates a sparse depth rendering of the point cloud represented by the input image and input key frames. The resulting reciprocal depth estimates are then rescaled pixel-by-pixel using a reduced resolution version of the input image and a reduced resolution version of each input frame.

[0130] The stereoscopic transformation engine 124 performs disparity estimation on the reduced resolution version of the input image and on the reciprocal depth estimate associated with each pixel of the reduced resolution version of the input image. As already described with respect to FIG. 3B, the stereoscopic transformation engine 124 aligns the per-pixel reciprocal depth information for the reduced resolution version of the input image to produce a normalized expected disparity value for each pixel. In various other embodiments, the user provides the stereoscopic transformation engine 124 with a reference disparity in the form of a manual annotation of one or more portions of the input image 350 selected by the user. The stereoscopic transformation engine 124 aligns the expected disparity to the provided reference disparity value, for example, using a least squares alignment technique. The stereoscopic transformation engine 124 performs disparity estimation for each input keyframe in a similar manner. For the input image and each input keyframe, the stereoscopic transformation engine 124 upsamples the associated low-resolution disparity estimate. The stereoscopic transformation engine 124 generates, for the input image and each input keyframe, a predicted disparity map by associating the upsampled disparity estimate with the corresponding high resolution input image or keyframe.

[0131] In operation 706, the stereoscopic transformation engine 124 generates motion vectors to map the input keyframe (in one embodiment, TIFF2024169343000025.tif7114

[0132] ) the expected output image TIFF2024169343000026.tif6114

[0133] In some embodiments, the stereoscopic transformation engine 124 warps the forward vector to the input keyframe TIFF2024169343000027.tif6114

[0134] and the input image TIFF2024169343000028.tif6114

[0135] The stereo transformation engine 124 generates a disparity map based on a forward optical flow between time instance t and time instance t-1, and a disparity value for time instance t from the predicted disparity map. The forward optical flow includes a pixel-by-pixel vector that represents the motion of objects depicted in the scene between time instance t-1 and time instance t. The stereo transformation engine 124 further reverse warps the disparity value for time instance t to time instance t-1. The stereo transformation engine 124 then adds the warped disparity value to the optical flow vector to obtain TIFF2024169343000029.tif6114

[0136] from TIFF2024169343000030.tif6114

[0137] Resulting in a forward vector to

[0138] In another embodiment, the stereoscopic transformation engine 124 converts the inverse vector into TIFF2024169343000031.tif7114

[0139] and input keyframe TIFF2024169343000032.tif7114

[0140] The stereo transformation engine 124 then generates a stereo image based on the backward optical flow between the frames, which includes pixel-by-pixel vectors that represent the motion of objects depicted in the scene between time instance t and time instance t-1. TIFF2024169343000033.tif6114

[0141] The stereoscopic transformation engine 124 also generates reverse (right-to-left) disparity values ​​by running the stereoscopic transformation engine 124 on the input image as already described for Figure 3A to calculate the negative of the warped per-pixel disparity values ​​obtained after inpainting 335 of Figure 3A. The stereoscopic transformation engine 124 then adds the warped right-to-left disparity values ​​to the reverse optical flow to produce a reverse vector.

[0142] In yet another embodiment, the stereoscopic transformation engine 124 generates an output image generated from a single input image as previously described with respect to FIG. 3A. TIFF2024169343000034.tif6114

[0143] Based on the estimate of , we generate a direct predicted vector. TIFF2024169343000035.tif6114

[0144] Given the estimate of TIFF2024169343000036.tif6114

[0145] and TIFF2024169343000037.tif7114

[0146] The optical flow between the two images is directly estimated, and the forward or backward vector is calculated.

[0147] In operation 708, the stereo transformation engine 124 performs pyramidal feature extraction on the input image and each input keyframe at various levels of resolution. For each level of resolution, the stereo transformation engine 124 generates a feature vector that represents the input image or input keyframe at that resolution. The multiple feature vectors together form a multi-resolution feature vector that represents the input image or input keyframe in a feature space.

[0148] The stereoscopic transformation engine 124 executes a second set of trained neural networks to warp the feature space representation of the input image with the motion vectors generated as described above. Specifically, each pixel of the feature space representation of the input image is shifted by the value of the generated motion vector. In various embodiments that include multiple input keyframes, a motion vector is generated between each input keyframe and the input image. The stereoscopic transformation engine 124 then warps the feature space representation of the input image based on each of these multiple motion vectors.

[0149] As already described with respect to FIG. 6, multiple pixels may be warped (splatted) to the same location in the feature space representation of the input image. The stereo transformation engine 124 approaches these occluded pixels using a differentiable depth-aware warping strategy that includes weights that determine the order of multiple pixels warped to the same location. This weighted order is determined from the estimated disparity values ​​for each splatted pixel and determines which pixels will remain visible in the expected output image. The stereo transformation engine 124 performs the above warping and splatting process using a full-level multi-resolution feature vector representation of the input image. For each resolution level, the stereo transformation engine 124 upsamples the estimated disparity values ​​by interpolation to match the particular resolution level.

[0150] In operation 710, the stereo transformation engine 124 performs background-aware inpainting and synthetic refinement using the features and disparity of the warped images for multiple resolutions. The inpainting and synthetic refinement complements previously hidden but newly disoccluded background regions and reduces aliasing and sharp edges at new boundaries between foreground objects and background in occluded regions. The stereo transformation engine 124 performs background-aware inpainting using local multi-scale self-attention as previously described with respect to FIG. 3A. Additionally, the stereo transformation engine 124 may analyze the input images and input keyframes to find background regions that were previously visible in one or more input keyframes but are occluded in the input images, and use these regions to refine the inpaint. For example, a particular section of the background may be visible in an input keyframe representing time instance t-1, but is occluded in an input image representing time instance t. The stereoscopic transformation engine 124 may determine that particular section of the background as newly disoccluded in the expected output image and use the background information from the input keyframe representing time instance t-1 to more accurately inpaint the newly disoccluded area.

[0151] In operation 712, after warping, background-aware inpainting, and synthetic refinement, the stereoscopic transformation engine 124 transforms the multi-resolution feature space representation of the input images and input keyframes into the final expected output images.

[0152] In short, the stereo transformation engine processes input images of video content to generate corresponding output images, such that the input images and the corresponding output images form a stereoscopic representation of the content of the input images. All functions and transformations used by the stereo transformation engine are fully differentiable, helping to train the underlying stereo transformation model end-to-end by backpropagation. In operation, the stereo transformation engine performs disparity estimation on the input images by calculating the inverse per-pixel depth for a reduced resolution version of the input frame, and then performing an alignment step to yield disparity values, i.e., point-wise displacement vectors of pixels. When calculating disparity values ​​for the input images, a range of possible disparity values ​​is provided and the calculated values ​​are normalized to be within this range, or a reference disparity is used to adjust the calculated values. The reference disparity can be in the form of a ground truth disparity (used during training) or a disparity value proposed by the user and provided at inference time. The user may also provide further aesthetic control inputs, such as selecting important objects in the input frames, identifying points in the input frames that correspond to stationary background regions, or providing camera parameters or camera placement information. The calculated and adjusted / normalized disparity values ​​are then used to generate a high-resolution disparity estimate. The stereo transformation engine further extracts features from the input frames at various resolutions.

[0153] The stereo transformation engine performs feature warping, depth-aware splatting, and inpainting in feature space based on the extracted features and high-resolution disparity estimation. Splatting refers to mapping (warping) multiple pixels in the feature space representation of the input image to the same pixel with its associated displacement vector. Splatting is depth-aware because the stereo transformation engine determines which splatted pixels should appear in the foreground of the predicted target frame based on the estimated disparity values. The stereo transformation engine performs background-aware inpainting on disoccluded regions of the feature space representation of the input image. These disoccluded regions are filled in using only pixels and textures from the image background. The stereo transformation engine generates a predicted output image from the warped, splatted, and inpainted feature space image representation.

[0154] The above process for generating a single expected output image from a single input frame can be extended to accept multiple input images and generate multiple expected output images. Information contained in multiple adjacent, nearby, or near-nearby input images is used to guide the stereoscopic transformation process, allowing the stereoscopic transformation engine to increase temporal consistency across the multiple expected output images, improving the illusion of depth perceived by a viewer in a stereoscopic pair comprising an input image and a expected output image.

[0155] One technical advantage of the disclosed technique over the prior art is that the functions and transformations used in the various stereoscopic transformation steps are fully differentiable, allowing for improved end-to-end training of the disclosed stereoscopic transformation model. Additionally, the disclosed technique allows for arbitrary aesthetic control input from the user, as well as the option to consider additional inputs beyond the input frames. Additionally, the disclosed technique incorporates expected disparity from input images for depth-aware splatting and inpainting, and uses disparity information from known reference input frames when generating and synthesizing corresponding output images. The use of an optional large number of input images also allows the disclosed technique to increase temporal consistency across multiple expected output video images. These technical advantages provide one or more technical improvements over prior approaches.

[0156] 1. In various embodiments, a computer-implemented method includes generating a predicted disparity map for an input image based on one or more depth values ​​associated with pixels in the input image using one or more machine learning models; extracting from the input image feature data representing pixels in the input image; applying an offset to each pixel represented by the feature data based on the predicted disparity map; identifying a plurality of offset pixels represented by the feature data that are offset to the same pixel location; assigning a visibility value to each of the plurality of offset pixels based on the predicted disparity map; modifying each of the plurality of offset pixels based on the visibility value such that one pixel of the plurality of offset pixels is visible and the other plurality of offset pixels are hidden; and generating an output image based on the feature data and the modified plurality of offset pixels such that the input image and the output image form a stereoscopic pair of images.

[0157] 2. The computer-implemented method of claim 1, wherein the step of generating the expected disparity map includes the steps of generating a reduced resolution version of the input image, assigning per-pixel depth values ​​to pixels representing foreground and background objects depicted in the reduced resolution version of the input image, assigning per-pixel expected disparity values ​​in the disparity map based on the depth values ​​so that pixels representing foreground objects in the expected disparity map are assigned a higher disparity value than pixels representing background objects, and upsampling per-pixel disparity map values ​​associated with the reduced resolution version of the input image to the original resolution of the input image.

[0158] 3. The computer-implemented method of claim 1 or 2, further comprising the steps of accepting one or more user annotations of the input image, each user annotation including an identification of a selected portion of the input image and a ground truth disparity value associated with the selected portion, and aligning the predicted disparity values ​​for each pixel for the selected portion of the input image with the associated ground truth disparity values ​​for the selected portion of the input image.

[0159] 4. The computer-implemented method of any one of items 1 to 3, wherein each of the one or more user annotations of the input image includes at least one of a bounding box surrounding a portion of the input image, one or more brush strokes indicating a portion of the input image, or a manual selection of an object depicted in the input image.

[0160] 5. The computer-implemented method of any one of items 1 to 4, wherein the step of modifying each of the plurality of offset pixels includes the steps of comparing associated disparity map values ​​for the plurality of offset pixels, and rendering one of the plurality of offset pixels visible based on the one of the plurality of offset pixels having the highest disparity map value, while hiding a pixel at the same pixel location having a lower disparity map value.

[0161] 6. The computer-implemented method of any one of items 1 to 5, wherein all functions and transformations performed by the one or more machine learning models are fully differentiable.

[0162] 7. The computer-implemented method of any one of items 1 to 6, further comprising the steps of: generating a disocclusion mask representing pixel locations in the feature data where portions of the image background are disoccluded after applying an offset to each offset pixel represented by the extracted feature data; complementing the pixel locations represented by the disocclusion mask with image data selected from the image background; and generating an output image based on the modified offset pixels and the pixel locations represented by the complemented disocclusion mask.

[0163] 8. The computer-implemented method of any one of items 1 to 7, wherein the image data used to complete the pixel locations represented by the disocclusion mask is selected by applying a neighborhood-based self-attention technique.

[0164] 9. The computer-implemented method of any one of items 1 to 8, wherein the input image is a single frame of a video sequence, the method further comprising the steps of: accepting as input one or more keyframes, each keyframe comprising a frame from the video sequence located at a temporal distance from the input image; generating a predicted disparity map for each keyframe based on one or more depth values ​​associated with pixels in the keyframe; creating motion vectors between the input image and each keyframe based on the predicted disparity map for each keyframe that describe changes in position of objects depicted in the keyframes and the input image; extracting keyframe feature data from each keyframe; and applying the motion vectors to the extracted keyframe feature data for each keyframe to derive a step of complementing pixel locations represented by the disocclusion mask.

[0165] 10. The computer-implemented method of any one of items 1 to 9, wherein a plurality of keyframes are accepted as input, the plurality of keyframes being selected at exponentially increasing temporal distances from the input image.

[0166] 11. In various embodiments, one or more non-transitory computer readable media store instructions that, when executed by one or more processors, cause the one or more processors to: generate a predicted disparity map for an input image based on one or more depth values ​​associated with pixels in the input image using one or more machine learning models; extract from the input image feature data representing pixels in the input image; apply an offset to each pixel represented by the feature data based on the predicted disparity map; identify a plurality of offset pixels represented by the feature data that are offset to the same pixel location; assign a visibility value to each of the plurality of offset pixels based on the predicted disparity map; modify each of the plurality of offset pixels based on the visibility value such that one pixel of the plurality of offset pixels is visible and the other plurality of offset pixels are hidden; and generate an output image based on the feature data and the modified plurality of offset pixels such that the input image and the output image form a stereoscopic pair of images.

[0167] 12. The one or more non-transitory computer-readable media of item 11, wherein the instructions further cause the one or more processing units to perform the steps of generating a reduced resolution version of the input image, assigning per-pixel depth values ​​to pixels representing foreground and background objects depicted in the reduced resolution version of the input image, assigning predicted per-pixel disparity values ​​in a disparity map based on the depth values ​​so that pixels representing foreground objects are assigned higher disparity values ​​in the predicted disparity map than pixels representing background objects, and upsampling per-pixel disparity map values ​​associated with the reduced resolution version of the input image to an original resolution of the input image.

[0168] 13. One or more non-transitory computer-readable media of item 11 or 12, further comprising instructions to cause the one or more processing units to perform steps of accepting one or more user annotations of the input image, each user annotation including an identification of a selected portion of the input image and a ground truth disparity value associated with the selected portion, and aligning a pixel-by-pixel predicted disparity value for the selected portion of the input image with the associated ground truth disparity value for the selected portion of the input image.

[0169] 14. One or more non-transitory computer-readable media described in any one of items 11 to 13, wherein the instructions for modifying each of the plurality of offset pixels further cause the one or more processing units to perform the steps of comparing associated disparity map values ​​for the plurality of offset pixels and rendering one of the plurality of offset pixels to be visible based on the one of the plurality of offset pixels having the highest disparity map value, while hiding a pixel at the same pixel location having a lower disparity map value.

[0170] 15. One or more non-transitory computer-readable media described in any one of items 11 to 14, wherein the instructions further cause the one or more processing units to perform the steps of generating a disocclusion mask representing pixel locations in the feature data where portions of the image background are disoccluded after applying an offset to each offset pixel represented by the extracted feature data, complementing the pixel locations represented by the disocclusion mask with image data selected from the image background, and generating an output image based on the modified plurality of offset pixels and the pixel locations represented by the complemented disocclusion mask.

[0171] 16. One or more non-transitory computer-readable media according to any one of items 11 to 15, wherein the input image is a single frame of a video sequence, and the instructions further cause the one or more processing units to perform the steps of accepting as input one or more keyframes, each keyframe comprising a frame from the video sequence located at a temporal distance from the input image; generating a predicted disparity map for each keyframe based on one or more depth values ​​associated with pixels in the keyframe; creating motion vectors between the input image and each keyframe based on the predicted disparity map for each keyframe that describe changes in position of objects depicted in the keyframes and the input image; extracting keyframe feature data from each keyframe; and applying the motion vectors to the extracted keyframe feature data for each keyframe to derive a step of complementing pixel locations represented by a disocclusion mask.

[0172] 17. In various embodiments, a computer-implemented method includes running a first neural network to predict per-pixel depth and disparity values ​​associated with an object depicted in an input image; updating parameters of the first neural network based on a first calculated training loss associated with the predicted disparity values; repeatedly running the first neural network to update the parameters of the first neural network; terminating execution of the first neural network if the first calculated training loss is less than a first threshold; sequentially running the first neural network and a second set of neural networks while holding the parameters of the first neural network constant; updating parameters of the second set of neural networks based on a second calculated training loss associated with a predicted output image; and updating parameters of the second set of neural networks while updating the parameters of the second set of neural networks. The method includes the steps of starting a first iterative sequential execution of the first neural network and the second set of neural networks, terminating the first iterative sequential execution of the first neural network and the second set of neural networks when the second calculated training loss is less than a second threshold, sequentially executing the first neural network and the second set of neural networks, updating the first neural network and the second set of neural networks based on a third calculated training loss associated with the expected output image, starting a second iterative sequential execution of the first neural network and the second set of neural networks while updating parameters of the first neural network and the second set of neural networks, and terminating the second iterative sequential execution of the first neural network and the second set of neural networks when the third calculated training loss is less than a third threshold.

[0173] 18. The first computed training loss is the disparity loss L which indicates the distance L1 between the expected disparity values ​​from the first neural network and the known training dataset disparity values. dItem 18. The computer-implemented method according to item 17,

[0174] 19. The second calculated training loss is a) a frame loss L that represents the distance L1 determined from a pixel-by-pixel comparison of the expected output image to the ground truth image versus the training data. L1 and b) the perceptual loss L, which represents the difference between the expected output image of the high-level semantic features and the ground truth image vs. the training data. F 19. The computer-implemented method according to item 17 or 18, comprising at least one of:

[0175] 20. The third calculated training loss is L = L L1 +γ L F +δ L d where γ and δ are the perceptual losses L F and the disparity loss L d 20. The computer-implemented method of any one of items 17 to 19, wherein the weight parameters associated with the

[0176] Any and all combinations of any claim element recited in any claim and / or any element described herein, in any aspect, are within the scope of the invention and its intended protection.

[0177] The description of various embodiments has been provided by way of example and is not intended to be exhaustive or to limit the invention to the disclosed embodiments. Numerous modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

[0178] Aspects of embodiments of the present invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware, all of which may be generally referred to herein as a "module," "system," or "computer." Furthermore, any hardware and / or software technique, process, function, component, engine, module, or system described in this disclosure may be implemented as a circuit or set of circuits. Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code thereon.

[0179] Any combination of one or more computer readable media may be used. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples (non-exclusive list) of computer readable storage media include an electrical connection including one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the context of this specification, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0180] Aspects of the present disclosure are described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine. The instructions, when executed by the computer or other programmable data processing apparatus, cause the computer or other programmable data processing apparatus to perform the functions / acts indicated in the flowchart and / or block diagram blocks. Such a processor may be, but is not limited to, a general purpose processor, a special purpose processor, an application specific processor, or a field programmable gate array.

[0181] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of implementing the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams represents a module, segment, or portion of code, which may include one or more executable instructions for performing a particular logical function. It should also be noted that in some alternative embodiments, the functions shown in the blocks may be performed in a different order than that shown in the drawings. For example, depending on the functionality involved, two consecutively shown blocks may in fact be executed substantially simultaneously, or in some cases, the blocks may be executed in the reverse order. It will also be understood that each block of the block diagrams and / or flowcharts, as well as combinations of multiple blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system or a combination of dedicated hardware and computer instructions for performing a particular function or act.

[0182] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, the scope of which is defined by the appended claims. [Explanation of symbols]

[0183] 100 Computing equipment 122 Training Engine 124 Stereoscopic Conversion Engine 220 Training Data Preprocessing Engine 230 Machine Learning Models 235 Disparity Estimation Network 240 Conversion Network 242 Feature Extraction Network 244 Warping and Splatting Network 246 Image Synthesis Network

Claims

1. 1. A computer-implemented method for performing stereoscopic transformation, comprising: generating a predicted disparity map for an input image based on one or more depth values ​​associated with pixels in the input image using one or more machine learning models; extracting feature data from the input image that is representative of the pixels in the input image; applying an offset to each pixel represented by the feature data based on the expected disparity map; identifying a plurality of offset pixels represented by the feature data, the offset pixels being offset to the same pixel location; assigning a visibility value to each of the offset pixels based on the predicted disparity map; modifying each of the plurality of offset pixels based on the visibility value such that one pixel of the plurality of offset pixels is visible and other pixels of the plurality of offset pixels are hidden; generating an output image based on the feature data and the modified plurality of offset pixels such that the input image and the output image form a stereoscopic pair of images; A computer-based method comprising:

2. The step of generating a predicted disparity map comprises: generating a reduced resolution version of the input image; assigning per-pixel depth values ​​to pixels representing foreground and background objects depicted in the reduced resolution version of the input image; assigning an expected disparity value for each pixel in the disparity map based on the depth value, such that pixels representing foreground objects in the expected disparity map are assigned a higher disparity value than pixels representing background objects; upsampling the disparity map values ​​for each pixel associated with the reduced resolution version of the input image to the original resolution of the input image; 2. The computer-implemented method of claim 1, comprising:

3. receiving one or more user annotations of the input image, each user annotation including an identification of a selected portion of the input image and a ground truth disparity value associated with the selected portion; aligning the predicted disparity values ​​for each pixel for the selected portion of the input image with the associated ground truth disparity values ​​for the selected portion of the input image; The computer-implemented method of claim 2 further comprising:

4. 4. The computer-implemented method of claim 3, wherein each of the one or more user annotations of the input image comprises at least one of a bounding box surrounding a portion of the input image, one or more brush strokes indicating a portion of the input image, or a manual selection of an object depicted in the input image.

5. The step of modifying each of the plurality of offset pixels comprises: comparing the disparity map values ​​associated with the offset pixels; rendering the one of the plurality of offset pixels as visible based on the one of the plurality of offset pixels having a highest disparity map value while obscuring pixels at the same pixel location having lower disparity map values; 2. The computer-implemented method of claim 1, comprising:

6. 10. The computer-implemented method of claim 1, wherein all functions and transformations performed by the one or more machine learning models are fully differentiable.

7. generating a disocclusion mask representing pixel locations in the extracted feature data where portions of the image background are disoccluded after applying the offset to each offset pixel represented by the extracted feature data; complementing the pixel locations represented by the disocclusion mask with image data selected from the image background; generating the output image based on the modified plurality of offset pixels and pixel locations represented by the interpolated disocclusion mask; The computer-implemented method of claim 1 further comprising:

8. 8. The computer-implemented method of claim 7, wherein the image data used to complete pixel locations represented by the disocclusion mask is selected by applying a neighborhood-based self-attention technique.

9. The input image is a single frame of a video sequence, the method further comprising: accepting as input one or more keyframes, each keyframe comprising a frame from the video sequence that is located a temporal distance from the input image; generating a predicted disparity map for each said keyframe based on one or more depth values ​​associated with pixels in that keyframe; generating motion vectors between the input image and each of the key frames that describe changes in position of objects depicted in the key frames and the input images based on a predicted disparity map for each of the key frames; extracting key frame feature data from each said key frame; applying the motion vectors to the extracted key frame feature data for each of the key frames to derive a interpolation of pixel locations represented by the disocclusion mask; 8. The computer-implemented method of claim 7, comprising:

10. 10. The computer-implemented method of claim 9, further comprising accepting as input a plurality of keyframes, the plurality of keyframes being selected at exponentially increasing temporal distances from the input image.

11. One or more non-transitory computer-readable media having instructions stored therein that, when executed by one or more processors, cause the one or more processors to: generating a predicted disparity map for an input image based on one or more depth values ​​associated with pixels in the input image using one or more machine learning models; extracting feature data from the input image that is representative of the pixels in the input image; applying an offset to each pixel represented by the feature data based on the expected disparity map; identifying a plurality of offset pixels represented by the feature data, the offset pixels being offset to the same pixel location; assigning a visibility value to each of the offset pixels based on the predicted disparity map; modifying each of the plurality of offset pixels based on the visibility value such that one pixel of the plurality of offset pixels is visible and other pixels of the plurality of offset pixels are hidden; generating an output image based on the feature data and the modified plurality of offset pixels such that the input image and the output image form a stereoscopic pair of images; One or more non-transitory computer-readable media that cause

12. The instructions may further cause the one or more processors to: generating a reduced resolution version of the input image; assigning per-pixel depth values ​​to pixels representing foreground and background objects depicted in the reduced resolution version of the input image; assigning an expected disparity value for each pixel in the disparity map based on the depth value, such that pixels representing foreground objects in the expected disparity map are assigned a higher disparity value than pixels representing background objects; upsampling the disparity map values ​​for each pixel associated with the reduced resolution version of the input image to the original resolution of the input image; 12. The one or more non-transitory computer-readable media of claim 11,

13. The instructions may further cause the one or more processors to: receiving one or more user annotations of the input image, each user annotation including an identification of a selected portion of the input image and a ground truth disparity value associated with the selected portion; aligning the predicted disparity values ​​for each pixel for the selected portion of the input image with the associated ground truth disparity values ​​for the selected portion of the input image; 13. The one or more non-transitory computer-readable media of claim 12,

14. The instructions for modifying each of the plurality of offset pixels may further include causing the one or more processors to: comparing values ​​of the associated disparity map for the offset pixels; rendering the one of the plurality of offset pixels as visible based on the one of the plurality of offset pixels having a highest disparity map value while obscuring pixels at the same pixel location having lower disparity map values; 12. The one or more non-transitory computer-readable media of claim 11,

15. The instructions may further cause the one or more processors to: generating a disocclusion mask representing pixel locations in the extracted feature data where portions of the image background are disoccluded after applying the offset to each offset pixel represented by the extracted feature data; complementing the pixel locations represented by the disocclusion mask with image data selected from the image background; generating the output image based on the modified plurality of offset pixels and pixel locations represented by the interpolated disocclusion mask; 12. The one or more non-transitory computer-readable media of claim 11,

16. the input image is a single frame of a video sequence; The instructions may further cause the one or more processors to: accepting as input one or more keyframes, each keyframe comprising a frame from the video sequence that is located a temporal distance from the input image; generating a predicted disparity map for each said keyframe based on one or more depth values ​​associated with pixels in that keyframe; generating motion vectors between the input image and each of the key frames, the motion vectors describing changes in position of objects depicted in the key frames and the input images, based on the expected disparity map for each of the key frames; extracting key frame feature data from each said key frame; applying the motion vectors to the extracted keyframe feature data for each of the keyframes to derive a step of interpolating pixel locations represented by the disocclusion mask; 20. The one or more non-transitory computer-readable media of claim 15,

17. 1. A computer-implemented method for training a machine learning model to perform a stereoscopic transformation, comprising: running a first neural network that predicts pixel-by-pixel depth and disparity values ​​associated with objects depicted in the input image; updating parameters of the first neural network based on a first calculated training loss associated with the expected disparity value; repeatedly running the first neural network to update the parameters of the first neural network; terminating execution of the first neural network if the first calculated training loss is less than a first threshold; sequentially running the first neural network and a second set of neural networks while holding the parameters of the first neural network constant; updating parameters of the second set of neural networks based on a second calculated training loss associated with the expected output images; commencing a first iterative sequential execution of the first neural network and the second set of neural networks while updating the parameters of the second set of neural networks; terminating the first iterative sequential execution of the first neural network and the second set of neural networks if the second calculated training loss is less than a second threshold; sequentially executing the first neural network and the second set of neural networks; updating the first neural network and the second set of neural networks based on a third calculated training loss associated with the expected output images; commencing a second iterative sequential execution of the first neural network and the second set of neural networks while updating the parameters of the first neural network and the second set of neural networks; terminating the second iterative sequential execution of the first neural network and the second set of neural networks if the third calculated training loss is less than a third threshold. A computer-based method comprising:

18. The first calculated training loss is a disparity loss L that indicates the distance L1 between the expected disparity values ​​from the first neural network and the known training dataset disparity values. ｄ 20. The computer-implemented method of claim 17, wherein:

19. The second calculated training loss is a) A frame loss L representing the distance L1 determined from a pixel-by-pixel comparison of the expected output image to the ground truth image versus the training data. Ｌ１ and, b) A perceptual loss L that represents the difference between the expected output image and the ground truth image versus training data for high-level semantic features. Ｆ and 20. The computer-implemented method of claim 18, comprising at least one of:

20. The third calculated training loss is L=L Ｌ１ +γ・L Ｆ + δ·L ｄ where γ, δ are the perceptual losses L, respectively. Ｆ and the parallax loss L ｄ 20. The computer-implemented method of claim 19, wherein the weight parameters are trainable and associated with

Images