Occlusion-aware forward warp for video frame interpolation

Abstract

The systems and techniques described herein relate to generating inter-frame frames from a first frame and a second frame. An apparatus includes: a memory storing the first frame and the second frame; and a processor coupled to the memory and configured to: estimate at least one optical flow between the first frame and the second frame; generate at least one occlusion mask based on the at least one optical flow; generate at least one weighted mask based on the at least one optical flow and the at least one occlusion mask; generate at least one inter-frame optical flow based on the at least one optical flow and the at least one weighted mask; generate at least one warped frame based on at least one of the first frame or the second frame and the at least one inter-frame optical flow; and generate an inter-frame frame based on the at least one warped frame.

Description

Technical Field

[0001] This disclosure relates to frame interpolation. For example, aspects of this disclosure relate to systems and techniques for providing occlusion-aware forward distortion when performing frame interpolation (e.g., video frame interpolation). Background Technology

[0002] Interpolation can be used to generate frames. For example, Video Frame Interpolation (VFI) can be used to generate inter-frame frames between a first frame at a first time and a second frame at a second time. Interpolated frames (e.g., those generated using the VFI algorithm) can be used in many different fields, such as video compression, computer vision, machine learning, etc. Summary of the Invention

[0003] Systems and techniques for providing occlusion-aware forward warping during frame interpolation (e.g., video frame interpolation). According to some aspects, an apparatus for processing one or more frames is provided. The apparatus includes: one or more memories configured to store a first frame and a second frame; and one or more processors coupled to the one or more memories and configured to: estimate at least one optical flow between the first frame and the second frame; generate at least one occlusion mask based on the at least one optical flow; generate at least one weighted mask based on the at least one optical flow and the at least one occlusion mask; generate at least one inter-frame optical flow based on the at least one optical flow and the at least one weighted mask; generate at least one warped frame based on at least one of the first frame or the second frame and the at least one inter-frame optical flow; and generate an inter-frame frame between the first frame and the second frame based on the at least one warped frame.

[0004] According to some aspects, a method for processing one or more frames is provided. The method includes: estimating at least one optical flow between a first frame and a second frame; generating at least one occlusion mask based on the at least one optical flow; generating at least one weighted mask based on the at least one optical flow and the at least one occlusion mask; generating at least one inter-frame optical flow based on the at least one optical flow and the at least one weighted mask; generating at least one distorted frame based on at least one of the first frame or the second frame and the at least one inter-frame optical flow; and generating an inter-frame frame between the first frame and the second frame based on the at least one distorted frame.

[0005] According to some aspects, a non-transitory computer-readable medium is provided, on which instructions are stored, which, when executed by one or more processors, configure one or more processors to: estimate at least one optical flow between a first frame and a second frame; generate at least one occlusion mask based on the at least one optical flow; generate at least one weighted mask based on the at least one optical flow and the at least one occlusion mask; generate at least one inter-frame optical flow based on the at least one optical flow and the at least one weighted mask; generate at least one distorted frame based on at least one of the first frame or the second frame and the at least one inter-frame optical flow; and generate an inter-frame frame between the first frame and the second frame based on the at least one distorted frame.

[0006] According to some aspects, an apparatus is provided, the apparatus comprising: means for estimating at least one optical flow between a first frame and a second frame; means for generating at least one occlusion mask based on at least one optical flow; means for generating at least one weighted mask based on at least one optical flow and at least one occlusion mask; means for generating at least one inter-frame optical flow based on at least one optical flow and at least one weighted mask; means for generating at least one distorted frame based on at least one of the first frame or the second frame and at least one inter-frame optical flow; and means for generating an inter-frame frame between the first frame and the second frame based on at least one distorted frame.

[0007] In some aspects, one or more of the devices described herein are, are a part of, and / or include: extended reality (XR) devices or systems (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), mobile devices or wireless communication devices (e.g., mobile phones or other mobile devices), wearable devices (e.g., connected watches or other wearable devices), cameras, personal computers, laptop computers, vehicles or computing devices or components of vehicles, server computers or server devices (e.g., edge or cloud-based servers, personal computers acting as server devices, mobile devices such as mobile phones acting as server devices, XR devices acting as server devices, vehicles acting as server devices, network routers, or other devices acting as server devices), another device, or combinations thereof. In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device described above may include one or more sensors (e.g., one or more inertial measurement units (IMUs) (such as one or more gyroscopes, one or more gyroscope testers, one or more accelerometers, any combination thereof) and / or other sensors).

[0008] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to define the scope of the claimed subject matter. This subject matter should be understood with reference to the appropriate portions of the entire specification, any or all drawings, and each claim.

[0009] The foregoing and other features and aspects will become more apparent from the following description, claims and accompanying drawings. Attached Figure Description

[0010] The exemplary aspects of this application are described in detail below with reference to the following figures: Figure 1 The use of video frame interpolation algorithms according to various aspects of this disclosure is illustrated; Figure 2 This is a conceptual diagram illustrating conventional algorithms for performing video frame interpolation according to various aspects of this disclosure; Figure 3 Methods for performing video frame interpolation according to various aspects of this disclosure are illustrated; Figure 4 This is a conceptual diagram illustrating a comparison of backward warp and forward warp in video frame interpolation according to various aspects of this disclosure; Figure 5 Examples illustrate how to resolve ambiguity in data when performing video frame interpolation according to various aspects of this disclosure; Figure 6 Examples of resolving blurriness in pixel values ​​when performing video frame interpolation according to various aspects of this disclosure are illustrated; Figure 7 Examples of resolving blurriness in pixel values ​​when performing video frame interpolation according to various aspects of this disclosure are illustrated; Figure 8 Examples of resolving missing pixel values ​​when performing video frame interpolation according to various aspects of this disclosure are illustrated; Figure 9 Examples of resolving missing pixel values ​​when performing video frame interpolation according to various aspects of this disclosure are illustrated; Figure 10 Examples of resolving missing pixel values ​​when performing video frame interpolation according to various aspects of this disclosure are illustrated; Figure 11 Examples of applying backward distortion and using occlusion information when performing video frame interpolation according to various aspects of this disclosure are illustrated; Figure 12 An example process for performing video frame interpolation according to various aspects of this disclosure is illustrated; Figure 13 This illustrates a problem where, for two images, one image cannot generate nonlinear motion, according to various aspects of this disclosure; Figure 14 Examples of secondary video interpolation based on various aspects of this disclosure are provided; Figure 15 Methods for video frame interpolation according to various aspects of this disclosure are illustrated; Figure 16 Solutions for video interpolation based on various aspects of this disclosure are illustrated; Figure 17 The process for performing video frame interpolation according to various aspects of this disclosure is illustrated; Figure 18 Examples of solutions for generating optical flow and intermediate frames given any t value therebetween are provided according to various aspects of this disclosure. Figure 19 Examples of solutions for generating optical flow and intermediate frames given several pairs of images and scattered images are illustrated according to various aspects of this disclosure; and Figure 20 Example computing systems based on various aspects of this disclosure are illustrated. Detailed Implementation

[0011] Certain aspects of this disclosure are provided below. Some of these aspects may be applied independently, and some may be applied in combination, as will be apparent to those skilled in the art. Specific details are set forth in the following description for purposes of explanation to provide a thorough understanding of the various aspects of this application. However, it will be apparent, however, that various aspects may be practiced without these specific details. The accompanying drawings and descriptions are not intended to be limiting.

[0012] The following description provides only exemplary aspects and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of the exemplary aspects will provide those skilled in the art with descriptions that can be used to implement the exemplary aspects. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the scope of this application as set forth in the appended claims.

[0013] As previously noted, interpolation can be used to generate frames. For example, video frame interpolation (VFI) refers to generating an inter-frame I between the first image or frame I0 at time 0 and the second image or frame I1 at time 1. t Frames I0 and I1 are time-series images. Inter-frame frame I... t Positioned between frames I0 and I1 (0 < t < 1) (between time 0 and time 1). In the context of video compression, instead of saving, storing, or sending all frame information for all three frames, inter-frame frames are generated via the VFI algorithm.

[0014] Interpolated frames (e.g., those generated using the VFI algorithm) can be used in many different fields, such as video compression, computer vision, machine learning, etc. In some examples, using the VFI algorithm to generate reconstructed inter-frame frames reduces the amount of residual (e.g., representing the difference) between the reconstructed inter-frame frame and its original version. By reducing the residual, the bit rate of the compressed residual can be reduced, and thus bandwidth can be saved when sending the compressed residual. In some examples, the VFI algorithm can be used to generate higher frame rate video. For example, the VFI algorithm can process 15 frames per second (FPS) video to generate inter-frame frames, resulting in 30 FPS video, 60 FPS video, or video with a higher frame rate.

[0015] Many VFI algorithms utilize backward warping to synthesize or generate inter-frames, such as inter-frame I. t However, because of inter-frame I t Invisible and difficult to obtain accurately, optical flow is a challenge. To obtain optical flow, such algorithms can rely on machine learning systems (e.g., neural networks) to learn to predict optical flow during training by utilizing a ground truth optical flow (e.g., ground truth inter-frames are available during training), such as from inter-frame I... t The optical flow (called the stream) of the frame (I0) up to time 0 ) and from inter-frame I t The optical flow (called the stream) of the frame (I1) at time 1 The performance of machine learning models (e.g., neural network models) can be heavily dependent on the training data. However, such machine learning systems may suffer from problems such as the test data differing from the training data (or training distribution) used to train the machine learning system. For example, when the test data differs from the training data or time... Optical flow illusion (predicting flow without intermediate images) may not work well when the distribution differs from the training distribution. Backward warp is easily computed, but inter-frame I-axis warping is problematic. t It does not exist when performing video frame interpolation (in this case, it is inter-frame I). t (Invisible).

[0016] In some cases, and By assuming motion in The relationship is estimated linearly. Two streams are used. and Synthesis I t Forward warping can be performed. Forward warping can be difficult to compute (difficult to map, ambiguous regions). In addition, forward warping may result in missing values ​​and / or ambiguous values.

[0017] The present disclosure describes systems and techniques for providing a forward warping-based solution for video frame interpolation. According to some aspects, the systems and techniques may perform occlusion-aware forward warping. Thus, the systems and techniques may resolve pixel value ambiguities without the need for depth information. For example, forward warping may be performed to generate an optical flow map from unseen intermediate frames (also referred to as inter-frame or center frames). Missing values may be determined for the forward warped optical flow map from the unseen intermediate frames. Occlusion handling and the optical flow map generated from the unseen intermediate frames allow the systems and techniques to synthesize intermediate frames. The systems and techniques may address existing problems with forward warping, such as data that resolves ambiguities in pixel values and determines missing values.

[0018] Figure 1 The use of a video frame interpolation algorithm 100 in accordance with aspects of the present disclosure is illustrated. A first image or frame I0 is shown at a first time 0, which has a portion of a car traveling on a curved street. A second image or frame I1 is shown at time 1, which has a more complete view of the car at a later time. A video frame interpolation (VFI) algorithm 102 is shown, which generates an interpolated frame I at a time between time 0 and time 1 or (0 < t < 1). t . Figure 1 A general framework of the problems and solutions disclosed herein is shown.

[0019] Figure 2 FIG. 200 is a conceptual diagram of an example system for performing video frame interpolation in accordance with aspects of the present disclosure. Conventional algorithms, such as an intermediate feature refinement network (IFRNet) 204 from the Conference on Computer Vision and Pattern Recognition (CVPR) 2022 and an all-to-many field transform (AMT) 206 for efficient frame interpolation from CVPR 2023 (collectively referred to as "VFI algorithms"), use frame prediction optical flow 202 represented as and . (Shown as Figure 2 in ). The IFRNet 204 and AMT 206 apply backward warping to synthesize frame . Since frame is not visible, it is difficult to obtain accurate optical flow. The VFI algorithms rely on neural networks to learn to predict flow and by leveraging the ground truth flow available during training. In some aspects, having a ground truth means that the ground truth is available during training. The challenge is that when performing inference, the system does not have and must use the trained neural network to infer what Typically, optical flow is predicted from time 0 to time 1. The VFI algorithm then applies warping using the original image and optical flow. After the warping process, the system merges the data, thus generating inter-frame frames. During inference, since the original I... t The image is used to predict the optical flow from time t to time 0. Optical flow from time t to time 1 Then distort the image.

[0020] IFRNet 204 is presented in general, and its architecture and loss function are illustrated. IFRNet 204 is an encoder-decoder based network that first extracts pyramid contextual features from the input frames using a shared encoder, and then gradually refines the bilateral intermediate flow field F through a coarse-to-fine decoder. t->0 F t->1 and intermediate features of reconstruction t This continues until the final output is produced. Besides the common image reconstruction loss L... r In addition, a new task-oriented flow distillation loss L was designed. d and feature space geometric consistency loss L g This allows for more efficient guidance of the feature alignment process to intermediate frame synthesis. See Kong, Lingtong, et al., “Ifrnet: Intermediate feature refine network for efficient frame interpolation,” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (2022), which is incorporated herein by reference.

[0021] AMT 206 is an architecture overview. Input frames are fed to a correlation encoder to extract features, which are then reused to construct the bidirectional correlation volume. A context encoder then extracts pyramid features from the visible frames and generates initial bidirectional flows and interpolated intermediate features. Next, the algorithm uses the bidirectional flows to retrieve bidirectional correlations to jointly update the flow field and intermediate features at each level. Finally, the algorithm generates multiple sets of flow fields, occlusion masks, and residuals based on coarse estimates used for interpolating intermediate frames. See: Li, Zhen, et al., “AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation,” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (2023), which is incorporated herein by reference.

[0022] Model performance is highly dependent on the training data. When the test data differs significantly from the training data, or... Unlike the training distribution, this optical flow illusion (i.e., predicting flow without intermediate images) will not work well.

[0023] Figure 3 A method for performing video frame interpolation according to various aspects of this disclosure is illustrated. It should be noted that the frame sequence 300 includes a first frame I0 and a first time 0, a second frame I1 and a second time 1, and an intermediate frame I at time t between time 0 and time 1. t . Figure 2 The conventional VFI algorithm shown uses I0 and I1 images to predict optical flow. This type of VFI algorithm uses backward warping to synthesize...

[0024] like Figure 3 As shown, optical flow can be estimated. For example, by assuming motion in and The relationship is linear. To use these two streams to synthesize I... t Forward warping is required.

[0025] Figure 4This is a conceptual diagram 400 illustrating a comparison of backward and forward warping in video frame interpolation according to various aspects of this disclosure. Pixel data 402 represents the optical flow of data from time 0 to time 1. Here, the initial frame I0 at time 0 has pixel data 0, 5, 0, 0, 0, and the final frame I1 at time 1 has pixel data 0, 0, 0, 5, 0. Optical flow from 0 to 1 Given the values ​​0, 2, 0, 0, 0, frame I can be calculated based on the pixel data. t However, when performing video frame interpolation, there is no intermediate image. Numbers represent pixels in the image, and for example, the movement of the number 5 from I0 to I1 represents the movement of pixels in the image from one position to the next over time. Pixel data 402 shows how some pixels do not change from time 0 to time 1, such as the "0" values ​​in the first, third, and fifth columns or data positions shown in pixel data 402.

[0026] When performing backward twist At that time, the goal was to obtain data from the second image I1 and optical flow data. Generate the first image or frame I0. The concept is a "backward" warp, meaning the flow is from time 0 to time 1, but the predictions made are based on the second frame I1 and the optical flow data. To determine the first frame I0. For backward warp. In frame I1 at time 1, the optical flow is 0, 2, 0, 0, 0. The pixel data used together (as example data) is 0, 0, 0, 5, 0 to generate This could represent the backward-distorted pixel data 405 of I1. The value "5" in the fourth column or position of frame I1 is transmitted via optical flow data through what is known as ghosting. Access 5 in the second column and 5 in the fourth column of the backward-distorted frame data 405. This can also be achieved by using optical flow data. The occlusion information related to the "0" in the fourth column is used to obtain the backward warped pixel data 405 in the fourth column. Note that the "0" values ​​in the first, third, and fifth columns are directly transferred as pixel values ​​to the backward warped pixel value (405) of 0, 5, 0, 5, 0.

[0027] Forward Twist 406 is difficult to compute. Forward warping involves using an initial frame I0 (0, 5, 0, 0, 0) and optical flow data. I1 is calculated using (0, 2, 0, 0, 0). In this case, when processing the data, the values ​​in the first, third, and fifth columns of the forward-distorted pixel data are obviously all "0". However, there is a missing value representing I0 in the second column of the forward-distorted pixel data 407, and there is a conflicting value in the fourth column regarding whether the value should be 0 or 5. Figure 4 This illustrates how computationally or mapping becomes difficult when forward warping is used in the case of blurred regions and missing values ​​in forward-warped pixel data 407. In some examples, mapping is also difficult if the flow is a fractional value. By applying the interpolation method disclosed herein, problems such as… Figure 4 The issue of missing and conflicting values ​​is shown.

[0028] Figure 5 Further examples of the use of the forward warp algorithm 500 are illustrated. The forward warp algorithm requires a relatively large amount of memory for computation. In some cases, Computational Unified Device Architecture (CUDA) decoding has been implemented to perform forward warps, which enables computing systems to leverage parallel computing techniques, such as using graphics processing units (GPUs) and traditional central processing units (CPUs) on the computing system.

[0029] Within each pixel, a method or algorithm finds all corresponding pixels and uses depth information to combine them to resolve ambiguities or conflicting values ​​(e.g., one object is in front of another object at the same 2D pixel location). In this case, single depth is less accurate and requires additional computation.

[0030] The following are the equations used for forward twisting:

[0031] Here, p refers to the target mesh, q refers to the source mesh, and f or F' refers to the optical flow. The first pixel data 502 shows the target mesh data p (1, 2, 3, 4, 5), the source mesh data q (1, 2, 3, 4, 5), the optical flow f (0, 2, 0, 0, 0), and the sum of q and f (1, 4, 3, 4, 5).

[0032] Grid u 504 represents the calculation of the difference between the target grid data p and the sum of q+f. In grid u 504, the first row is 0, 0-3, -2, -3, -4, generated by subtracting each corresponding value of q+f (which is 1, 4, 3, 4, 5) from the first value 1 of the target grid data p (1, 2, 3, 4, 5) to produce 0, -3, -2, -3, -4. The second row of grid u 504 represents the second value 2 of the target grid data p subtracting the corresponding value of q+f (which is 1, 4, 3, 4, 5) to produce 1, -2, -1, -2, -3, and so on, filling grid u 504. Using the values ​​of grid u 504, the above b(u) equation can be applied. The b(u) equation is used to determine which pixels have a correspondence between the corresponding target value and the corresponding source grid value. The B(u) grid 506 shows a value of 1 in the first row and first column, indicating the result of equation b(u) showing the correspondence between the target and source grids at that location. In the second row, there is no "1" value, thus indicating no correspondence. In the third row and third column, there are 1 values ​​indicating a correspondence. In the fourth row, second column, and fourth column, there are "1" values ​​indicating a conflicting correspondence. This method can use depth information to select one of these two values. The fifth row and fifth column contain 1 values ​​indicating a correspondence.

[0033] In the fourth row of B(u) grid 506, the method may include using a single depth estimation model to predict depth information. It can be assumed that objects closer to the camera should be more dominant. Therefore, the system predicts depth or inverse depth D1(1) D1(2) D1(3) D1(4) D1(5), thus obtaining objects closer with relatively higher values. In some respects, it is assumed that the value 5 in pixel data 508 represents a car in the foreground of the image. In this case, if the car moves to the right and other data (0') represent the background, the car will be closer to the camera than the background. The algorithm will give more weight to the value 5 representing the car. Therefore, in pixel data 508, the obtained I s The pixel data (forward-distorted image) still has no value in the second column because there is no correspondence indicated by the second row of B(u) grid 506. However, applying the above I... s (p) The equation solves I by using depth information and adding additional weights to the second column position of the fourth row of the B(u) grid 506. s The ambiguity in the fourth column (the 1 in the fourth row of the second column is weighted more than the 1 in the fourth row of the fourth column) thus provides the appropriate result. In other words, the value 5 in the second column of the I0 pixel data is given a higher weight, and therefore when 5 moves right from I0 to I1, then when it is in the fourth column, its value is given a higher weight and thus used for I... s The fourth column.

[0034] It should be noted that in the B(u) grid 506, the first row and first column have a 1 indicating the correspondence between the source grid and the target grid at that location. Therefore, in the pixel data 508, the data in the first column are 0 and 0 for I0 and I1 respectively, and the 0 value is thus carried to I. s The first column. The 1s in the third row and third column of B(u)506 indicate the correspondence between the third column of I0 and I1, and the 0 value is therefore carried to I. s The third column. Similarly, the 1s in the fifth row and fifth column of B(u)506 indicate the correspondence between the fifth column of I0 and I1, and the 0 value is therefore carried to I. s Fifth column. I s The unknown values ​​in the second column remain unchanged because there is no corresponding relationship for that column in B(u)506.

[0035] Since the fourth row of B(u)506 has two "1" values, therefore the above I s (p) The equation is operable to make occluded pixels dominate the analysis.

[0036] Performing single-depth calculations is less accurate and requires additional computation. Figure 6 An example of the proposed solution 600 is provided, which offers an occlusion-aware forward warping method. Pixels that are not occluded but cause occlusion of other pixels may be closer to the camera than occluded pixels. This method involves using an occlusion mask based on forward-backward consistency.

[0037] One-dimensional example 602 shows a first image I0 with values ​​0, 5, 0, 0, 0, and a second image with values ​​0, 0, 0, 5, 0, where the flow from I0 to I1... The values ​​are 0, 2, 0, 0, and the flow is from I1 to I0. The values ​​are 0, 0, 0, -2, and 0. I 0。 The value of "2" in the sequence 0, 2, 0, 0, 0 indicates that the value 5 has moved from the second position in I0 to the fourth position in I1. However, in the opposite direction... The values ​​are 0, 0, 0, -2, 0, where "-2" represents the value "5" moving from the fourth position to the second position in a negative or leftward direction. Therefore, in optical flow... In the middle, the corresponding value is -2.

[0038] To generate an occlusion mask, the following example equation can be used:

[0039] The occlusion mask O0 shown in pixel data 604 can be generated based on forward-backward consistency, as described in the following literature: Jeong, Jisoo et al., “Distractflow: Improving optical flow estimation via realistic distractions and pseudo-labeling,” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (2023), which is incorporated herein by reference. f and V b It can represent the predicted forward and backward flows, values 1 and 2 can represent constants such as 0.01 and 0.5, respectively. Other values ​​are also included within the scope of this disclosure. Generally, the method for the above equations is to determine whether the forward and backward flow predictions satisfy the constraints of the equations, and if so, it can be assumed that the predictions are accurate and the output can be 1. Otherwise, the output will be 0. By way of example, for each of the five positions, the value of "x" is the forward optical flow. or backward optical flow The corresponding values ​​of 0, 2, or -2 in the equation. This equation illustrates one example of how to determine forward-backward consistency, as other equations can be implemented to determine the consistency between two predictions.

[0040] The above equations also apply to forward flow. Backflow Both are used to obtain a set of values ​​for the occlusion mask O0 as values ​​0, 0, 0, 1, 0. Pixel data 604 is then shown in the following series of operations.

[0041] Forward flow It has values ​​0, 2, 0, 0, 0, in Figure 6 As shown in the diagram. Using forward flow. To apply back warp and use occlusion mask O0 606 to obtain This results in the values: 0, 1, 0, 1, 0. Note that backward warping is similar to... Figure 4 The backward twist 404 is shown. The system can then apply it pixel-by-pixel. To obtain the values ​​0, 1, 0, 0, 0. The values ​​0, 1, 0, 0, 0 indicate which position in the data should be weighted. The application indicates the use of a backward-distorted occlusion mask. During operation, the occlusion moves from the fourth column or fourth position to the second position. A weighted mask 608 can be generated and applied as... σ can be a value such as a hyperparameter or other value to obtain values ​​1, 1+σ, 1, 1, 1. Note that weighting is applied to the second position of the weighted mask 608. Since the occlusion indicates an object closer to the camera, its corresponding pixel value is preferably used in the distortion. Therefore, as shown, the pixel values ​​in the second position or column are given greater weight.

[0042] It should be noted that the position of the 1+σ value in the weighted mask 608 is based on the processing of the occlusion mask O0 606, which uses the value 1 to refer to the presence of occlusion at the corresponding position, and is then used through a backward warping process to identify which position should be given additional weight.

[0043] Figure 7 An example of resolving blurriness at pixel value 700 is illustrated. Pixel data 702 is shown as 0, 5, 0, 0, 0 for the first image I0, and as 0, 0, 0, 5, 0 for the second image, where the optical flow from I0 to I1... It is 0, 2, 0, 0, 0, and the optical flow from I1 to I0. It is 0, 0, 0, -2, 0. This shows the values ​​for... The weighted mask 706 has values ​​of 1, 1+σ, 1, 1, 1. The weighted mask 706 is used to resolve ambiguity when depth information is not required. In the B(u) grid 704, since the second position corresponds to an object closer to the camera, the system uses its value instead of the value in the fourth position. The system adds weights to I0(2) instead of I0(4). As indicated by the second column highlighted in B(u)704, the values ​​in the second column have the added weights of each weighted mask 706. Although the weighting of the entire column is shown in the second column of B(u)704, there is a conflict in the fourth row 705 of B(u)704, where there are two values ​​in that row. Since both values ​​are included in the fourth row 705, the weights are applied to the fourth row 705. Other rows do not exhibit this conflict (where there are two values ​​to choose from), in which case weighting is not required in those rows.

[0044] Figure 8 This illustrates a method for resolving the missing value 800. Forward warping can be applied to the optical flow map. Image representation 802 shows the first image I1 and the intermediate image I2. f The second image I0 shows the forward optical flow. Forward optical flow It is equal to or can be decomposed into the first optical flow from time 0 to time 1. (Multiplied by the value t) plus the second optical flow from time t to time 1 Some optical flow algorithms predict optical flow at 1 / 8 resolution and upsample it. Forward warping can be applied to optical flow at 1 / 8 resolution, which is very efficient. The original forward warping requires a large amount of memory. However, by performing optical forward flow at 1 / 8 resolution, the amount of memory required for forward warping can be greatly reduced.

[0045] In some respects, the disclosed methods can be implemented using matrix multiplication in PyTorch. Using matrix multiplication eliminates the need for a dedicated CUDA implementation.

[0046] When calculating the above Figure 8 When calculating the optical flow, the following equation can be used:

[0047] item It represents the distortion value during the process. (Item) This involves applying a forward warp operation to obtain the optical flow from time t to time 1, and based on the optical flow from time 0 to time 1. like Figure 8 As shown. This method is to... Decompose into different terms. The independent variable in the equation comes from the pixel movement from time 0 to time t. The above equation allows us to obtain... Then, it can be done by using calculate However, missing values ​​still need to be filled.

[0048] Pixel data 804 illustrates the various stages of calculating the optical flow, for It is 0, 2, 0, 0, 0; for It is 0, 1, 0, 0, 0; for (x) is 0, 1, 0, 0, 0, and for It is 0, 0, 1, 0, 0.

[0049] Figure 9 Examples include showing the first optical flow. Second optical flow A set of frames 902 and image data 900 are shown. These optical flows are in opposite directions as shown. If these two flows are combined, the result will simply be zero. A set of pixel data 904 (including pixel values) is also shown. Because... The optical flow map contains missing values, so due to forward warping, for each optical flow map, a missing pixel can be filled using a value from the other map. In this example, forward warping can be used to generate two optical flows. In this process, it can be assumed that... .

[0050] Figure 10 Examples are given for resolving missing values ​​in optical flow (e.g., Solutions for missing values ​​in the mesh. For example, when generating a forward warp using mesh B0(u) 1008, missing values ​​need to be filled. Provided from... Figure 9 The pixel data 904 is used as input to determine the missing values. Missing values ​​can be based on the following assumptions. To determine, for example, the pixel data of optical flow 1014 (represented as...) ) and the pixel data of optical flow 1016 (represented as As shown in the figure.

[0051] Assuming t=0.5, the method may include multiplying 0.5 by the optical flow of a set of pixels from pixel data 904. This produces pixel values ​​0, 1, 0, 0, 0 (1004). (1 - 0.5) is then multiplied by the optical flow of a set of pixels from pixel data 904. Pixel values ​​0, 0, 0, -1, 0 (1006) are generated. A first weighted mask 1010 is obtained based on pixel 1004. Note that the second position in the weighted mask 1010 has an additional weighting value σ. A second weighted mask 1013 can be determined based on pixel 1006. Note that the second weighted mask 1013 has an additional weighting value σ at the fourth position.

[0052] For mesh B0(u) 1008, given the first weighted mask 1010, the forward warping process is as follows. The first row and first column have a value of 1 (indicating correspondence), but for optical flow and pixel 1004, in optical flow... In the first position of 1014, the value will be zero. The second row and second column of B0(u) 1008 are both zero (thus meaning there is no correspondence), therefore the optical flow... The second position of 1014 will be missing. The third row of B0(u) 1008 has two conflicting values ​​of 1 in the second and third rows, therefore the weighted mask 1010 instructs to add weight to the second position, thereby affecting optical flow. In 1014, a value of 1 is generated at the third position. The fourth row of B0(u) 1008 includes 1 at the fourth position, which results in optical flow. The zeros in 1014 and the 1 in the fifth row and fifth column of B0(u) 1008 result in optical flow. The zero value in the fifth position of 1014.

[0053] A similar process to the forward twisting of B1(u) 1012 can be applied to use the second weighted mask 1013 to generate optical flow. 1016, where the missing value in the fourth position of the fourth row based on B1(u) 1012 indicates no corresponding relationship.

[0054] To fill in missing values, optical flow can be used. 1014, where the fourth position, which is 0, is used to generate optical flow. The value of the fourth position of 1016, using, for example (4) or The equation will be 0. In order to achieve optical flow... The second position of 1014 generates a missing value, which can be achieved using optical flow as follows. The second position of 1016: (2) could be – (-0.5 / 0.5) 0, the result is 0.

[0055] optical flow 1016 has values ​​of 0, 0, -1, 0, 0 and optical flow 1014 has values ​​0, 0, 1, 0, 0, which can now be used to generate inter-frame I. t . Figure 11 This example illustrates pixel data including frame I0 and optical flow. A diagram illustrating the fusion process 1100 that operates on the first set of datasets 1102 of occlusion mask O0 and 1016. Process 1100 includes applying a backward warping process and using occlusion information from occlusion mask O0 to fuse the two warped images to generate a final interpolated intermediate frame 1114 (or inter-frame frame or central frame).

[0056] The second set of data, 1106, includes pixel data for I1 and optical flow. 1014 and occlusion mask O1. Image I0 and optical flow. 1016 is used to generate a first backward-distorted image 1104 with values ​​0, 5, 5, 0, 0. Image I1 and optical flow Image 1014 is used to generate a second backward-distorted image 1108 with values ​​0, 0, 5, 5, 0. Occlusion mask O0 is used with the first backward-distorted image 1104 to generate a first set of pixels 1110 with values ​​0, 5, 0, 0 (where the second value is missing). Occlusion mask O1 is used with the second backward-distorted image 1108 to generate a second set of pixels 1112 with values ​​0, 0, 5, and 0 (where the fourth value is missing). Because occlusion mask O0 indicates a second position where the first set of pixels 1110 can be ignored, and occlusion mask O1 indicates a fourth position where the second set of pixels 1112 can be ignored, the system can merge the first set of pixels 1110 and the second set of pixels 1112 to achieve an inter-frame 1114 with values ​​0, 0, 5, 0, 0.

[0057] Figure 12 The overall pipeline of system 1200 is illustrated. System 1200 receives a first frame I0 1202 and a second frame I1 1204 as input. The optical flow from 0 to 1 and from 1 to 0 are estimated as follows: Optical flow generation can be performed using software models such as Recursive Full-Pair Transform (RAFT) or other models for estimating optical flow. The occlusion mask generator 1206 is used for... Occlusion masks are generated to produce O0 and O1. As described above, the occlusion mask generator 1206 can utilize a forward-backward consistency algorithm during the generation of the occlusion mask.

[0058] An optical mask O0 is provided to utilize the optical mask O0 and the first optical flow. The first weighted mask generator 1208 generates a first weighted mask W0. An optical mask O1 is provided using the optical mask O1 and optical flow. The second weighted mask generator 1210 generates the second weighted mask W1.

[0059] The first optical flow generator 1212 is based on the first weighted mask W0 and the first optical flow. Generate optical flow The second optical flow generator 1214 is based on the second weighted mask W1 and the second optical flow. Generate optical flow . optical flow The first frame I0 1202 is provided to the first backward warp and occlusion processing generator 1216 to generate the first backward warp image. Optical flow The first and second frames I1 1204 are provided to the second backward warp and occlusion processing generator 1218 to generate a second backward warp image. The first and second backward warp images are provided to the fusion engine 1220 to fuse the data and generate inter-frame I. t 1222.

[0060] Optionally, inter-frame I t 1222 can be further processed by neural network 1224, which can be trained to enhance inter-frame I. t 1222. For example, it is possible to base an inter-frame I with a reference truth value. t The training data was used to train the neural network 1224 to generate refined inter-frame I. t 1226.

[0061] On the other hand, there may be a portion of inter-frame 1222 where the confidence of the data covering the image region is low. In this case, the low-confidence region will be removed, and the neural network 1224 can be used to fill the removed region. Therefore, there are several ways to refine the image.

[0062] Figure 13 A graph 1300 illustrating various aspects of this disclosure is shown, illustrating that for two images I1 and I0, it is impossible to generate an inter-frame frame I. t This represents a nonlinear shift problem. This shift can be characterized by polynomial equations.

[0063] Figure 14 The graph 1400 is illustrated, in which quadratic video interpolation is used to seek a solution to the problem of generating nonlinear shifts between two images I1 and I0. Figure 14 The method shown is outlined in the following reference: Xu, Xiangyu et al., “Quadratic video interpolation,” Advances in Neural Information Processing Systems 32 (2019), which is incorporated herein by reference. Figure 14 Attempts to solve the problem included using a quadratic equation to determine the flow from time 0 to time 1, as follows:

[0064] Here, as Figure 14 As shown, the first optical flow It is shown as from I0 to I -1 And the second optical flow - It is the first optical flow The inverse of the equation. Using this equation, the system can learn from nonlinear movements.

[0065] Figure 15 Graph 1500 is shown as an example, illustrating how I can be obtained. t A set of expected data. Here, expectations project the problem as the need to determine different angles. and 'and the differences between these angles, to help determine the non-linear flow between images.'

[0066] Figure 16 Graph 1600 illustrates a solution for video interpolation. Here, the goal is to use angles... 1602a and The difference between '1604' and '1604' determines the optical flow and value of the inter-frame at point 1606. This represents the value along the x-axis from I0 to point 1608. This represents the y-value from point 1608 to point 1610. Note the angle. 1602a equals angle 1602b. If Then, this process can use the following equation to find the value at point 1606 in the inter-frame sequence. This means the angle... 1602a and The difference between '1604' and '1604' should be below a certain threshold. (See figure.) Including the following x and y values: . The value represents the inter-frame I at point 1606. i The corrected x and y values ​​for the position. In some respects, It is the length of line 1612. The value represents the length of line 1614. The value represents the length of line 1616. The values ​​include and , The above equation is used to determine the value of the inter-frame at point 1606, which is non-linear with respect to the first frame I0 and the second frame I1.

[0067] When assuming linear motion, the following formula can be applied. As previously described.

[0068] In the improved computation assuming nonlinear motion and using multiple frames, the following formula can be applied: ,in When considering multi-frame information, from arrive The estimated intermediate optical flow is as previously described.

[0069] Using the data obtained above, the optical flow can be determined using the corrected optical nonlinear optical flow, as shown below:

[0070] Figure 17This is a flowchart illustrating an example process 1700 for processing one or more data frames to generate inter-frame frames. Process 1700 may include any one or more steps disclosed herein. Process 1700 may be performed using a computing device (or apparatus) or components or systems of a computing device (e.g., a chipset, one or more processors, such as a plurality of CPUs, GPUs, NPUs, DSPs, etc.). The computing device may include, for example... Figure 12 The system 1200 for generating inter-frames includes an occlusion mask generator 1206, one or more weighted mask generators 1208, 1210, one or more optical flow generators 1212, 1214, one or more backward warping and occlusion processing generators 1216, 1218, a fusion engine 1220 and / or a neural network 1224, a computing system 2000, or a combination thereof.

[0071] At operation 1702, the computing device (or at least one component or system thereof) may estimate at least one optical flow between the first frame and the second frame. In some aspects, estimating at least one optical flow between the first frame and the second frame may include estimating a first optical flow in a forward direction from the first frame to the second frame, and estimating a second optical flow in a backward direction from the second frame to the first frame.

[0072] At operation 1704, the computing device (or at least one component or system thereof) may generate at least one occlusion mask based on at least one optical flow. In some aspects, generating at least one occlusion mask may include generating a first occlusion mask and a second occlusion mask based on a first optical flow and a second optical flow. The first occlusion mask and the second occlusion mask may be generated using an algorithm that determines the forward-backward consistency between the forward and backward optical flows.

[0073] At operation 1706, the computing device (or at least one component or system thereof) may generate at least one weighted mask based on at least one optical flow and at least one occlusion mask. In some aspects, generating at least one weighted mask may further include generating a first weighted mask based on a first optical flow and a first occlusion mask, and generating a second weighted mask based on a second optical flow and a second occlusion mask. In some aspects, at least one weighted mask may include weight values ​​based on occlusion values ​​in at least one occlusion mask.

[0074] At operation 1708, the computing device (or at least one component or system thereof) may generate at least one inter-frame optical flow based on at least one optical flow and at least one weighted mask. In some aspects, generating at least one inter-frame optical flow may further include generating a first inter-frame optical flow in a first direction based on a first optical flow and a first weighted mask, and generating a second inter-frame optical flow in a second direction based on a second optical flow and a second weighted mask.

[0075] At operation 1710, the computing device (or at least one component or system thereof) may generate at least one distorted frame based on at least one of the first frame or the second frame and at least one inter-frame optical flow.

[0076] In some aspects, in order to generate at least one distorted frame, a computing device (or at least one component or system thereof) may generate a first distorted frame based on a first frame and a first inter-frame optical flow in a first direction, and generate a second distorted frame based on a second frame and a second inter-frame optical flow in a second direction.

[0077] At operation 1712, the computing device (or at least one component or system thereof) may generate an inter-frame frame between the first frame and the second frame based on at least one warped frame.

[0078] In some aspects, a computing device (or at least one component or system thereof) may fuse a first distorted frame and a second distorted frame to generate an inter-frame frame. The computing device (or at least one component or system thereof) may be configured to, and may apply a neural network to, the inter-frame frame to generate a refined inter-frame frame. The neural network may be trained on training data with a baseline ground truth inter-frame frame.

[0079] In some aspects, an apparatus for processing one or more frames includes: one or more memories configured to store a first frame and a second frame; and one or more processors coupled to the one or more memories and configured to: estimate at least one optical flow between the first frame and the second frame; generate at least one occlusion mask based on the at least one optical flow; generate at least one weighted mask based on the at least one optical flow and the at least one occlusion mask; generate at least one inter-frame optical flow based on the at least one optical flow and the at least one weighted mask; generate at least one warped frame based on at least one of the first frame or the second frame and the at least one inter-frame optical flow; and generate an inter-frame frame between the first frame and the second frame based on the at least one warped frame.

[0080] In some respects, a non-transitory computer-readable medium (e.g., Figure 20 The memory 2015, ROM 2020, RAM 2025, or cache 2011 is configured to have instructions stored thereon that, when executed by one or more processors (e.g., processor 2012), configure one or more processors to: extract first prosodic data from input data; generate content embeddings based on the input data; extract second prosodic data from target speech; generate speaker embeddings from target speech; generate prosodic embeddings from the second prosodic data; and generate transformed prosodic data based on the first prosodic data and the prosodic embeddings.

[0081] In another aspect, an apparatus may include: components for extracting first prosodic data from input data; components for generating content embeddings based on the input data; components for extracting second prosodic data from target speech; components for generating speaker embeddings from the target speech; components for generating prosodic embeddings from the second prosodic data; and components for generating transformed prosodic data based on the first prosodic data and the prosodic embeddings. In some cases, the components for performing any of the functions described above may include... Figure 12 The system 1200 for generating inter-frames includes an occlusion mask generator 1206, one or more weighted mask generators 1208, 1210, one or more optical flow generators 1212, 1214, one or more backward warping and occlusion processing generators 1216, 1218, a fusion engine 1220 and / or a neural network 1224, a computing system 2000, or a combination thereof.

[0082] Computing devices may include any suitable device, such as mobile devices (e.g., mobile phones), desktop computing devices, tablet computing devices, XR devices (e.g., VR headsets, AR headsets, AR glasses, etc.), wearable devices (e.g., connected watches or smartwatches or other wearable devices), server computers, vehicles (e.g., autonomous vehicles) or computing devices of vehicles, robotic devices, laptop computers, smart TVs, cameras, and / or any other computing device with the resource capability to perform the processes described herein (including process 1700 and / or any other processes described herein). In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other types of data.

[0083] A component capable of implementing a computing device in a circuit. For example, the component may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein, and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0084] Process 1700 is illustrated as a logic flowchart, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or combinations thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which the operations are described is not intended to be construed as limiting, and any number of described operations can be combined in any order and / or in parallel to implement the process.

[0085] Additionally, process 1700 and / or any other process described herein may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors, by hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0086] Figure 18 Figure 1800 illustrates a system for enhancing training frame pairs by combining the image content of one frame with another image.

[0087] Figure 19 An example is given of a method for training an optical flow network using several pairs of data and scattered images 1900. By using the frame interpolation method described in this paper, interpolated intermediate frames can be generated, denoted as... In a given pair of consecutive frames and Optical flow maps between, and between the original frame and the intermediate frame, i.e. Such interpolated intermediate frames It can be used to significantly augment training data without requiring manual annotation. New training pipelines can leverage data augmentation techniques. Figure 19 The method shown involves generating four distinct pairs for each synthesized inter-frame to train the optical flow model. This method can then be further used... One of the scattered frames is as follows Figure 19 The scattered image is shown. This method involves applying the following process, which now provides four enhanced training pairs in addition to the original pairs (note that this is achieved through variations). (It can generate even more augmented samples)

[0088] Figure 20 These are illustrations illustrating examples of systems used to implement certain aspects of this disclosure. Specifically, Figure 20 An example of a computing system 2000 is illustrated. This computing system can be any computing device, such as a computing system, a camera system, or any component thereof, wherein the components of the system communicate with each other using connection 2005. Connection 2005 can be a physical connection using a bus, or a direct connection to a processor 2012, such as in a chipset architecture. Connection 2005 can also be a virtual connection, a networking connection, or a logical connection.

[0089] In some examples, the computing system 2000 is a distributed system, wherein the functions described in this disclosure may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some examples, one or more of the described system components represent a plurality of such components, each performing some or all of the functions targeted by the described components. In some examples, the components may be physical or virtual devices.

[0090] Example system 2000 includes at least one processing unit (CPU or processor) 2012 and a connection 2005 that couples various system components, including system memory 2015 (such as read-only memory (ROM) 2020 and random access memory (RAM) 2025), to processor 2012. Computing system 2000 may include a cache 2011 of high-speed memory that is directly connected to, closely proximate to, or integrated into processor 2012.

[0091] Processor 2012 may include any general-purpose processor and hardware or software services (such as services 2032, 2034, and 2036 stored in storage device 2030 and configured to control processor 2012), as well as dedicated processors in which software instructions are incorporated into the actual processor design. Processor 2012 may be a substantially completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0092] To enable user interaction, the computing system 2000 includes input devices 2045 that can represent any number of input mechanisms, such as microphones for voice, touchscreens for gesture or graphical input, keyboards, mice, motion input, and voice input. The computing system 2000 may also include output devices 2035 that can be one or more of a plurality of output mechanisms. In some instances, a multimodal system allows users to provide multiple types of input / output to communicate with the computing system 2000. The computing system 2000 may include a communication interface 2040, which typically controls and manages user input and system output.

[0093] The communication interface can perform or facilitate the receiving and / or transmitting of wired or wireless communications using wired and / or wireless transceivers, including utilizing audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple... ® Lightning ® Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, dedicated wired ports / plugs, Bluetooth ® Wireless signal transmission, Bluetooth ® Low-power (BLE) wireless signal transmission, IBEACON ® Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), microwave access global interoperability (WiMAX), infrared (IR) wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or those communications in some combination thereof.

[0094] The communication interface 2040 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers for determining the location of the computing system 2000 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the U.S. Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There are no limitations on operation on any particular hardware arrangement, and therefore the underlying features here can be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0095] Storage device 2030 may be a non-volatile and / or non-transitory and / or computer-readable storage device, and may be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as magnetic tape, flash memory cards, solid-state storage devices, digital multifunction disks, cartridges, floppy disks, hard disks, magnetic tapes, magnetic stripes, any other magnetic storage media, flash memory, memristor memory, any other solid-state storage, CD-ROM discs, rewritable CD discs, DVD discs, Blu-ray discs (BDD discs), holographic discs, another optical medium, secure digital (SD) cards, microSD cards, Memory Sticks. ® Cards, smart card chips, EMV chips, Subscriber Identity Module (SIM) cards, mini / micro / nano / micro SIM cards, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM / ReRAM), phase change memory (PCM), spin-transfer torque RAM (STT-RAM), another memory chip or cassette and / or combinations thereof.

[0096] Storage device 2030 may include software services, servers, services, etc., which enable the system to perform functions when the code defining such software is executed by processor 2012. In some examples, hardware services performing specific functions may include software components stored in computer-readable media connected to necessary hardware components, such as processor 2012, connection 2005, output device 2035, etc., to perform functions. The term "computer-readable media" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data can be stored and which does not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0097] In some respects, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0098] Specific details are provided in the foregoing description to provide a thorough understanding of the aspects and examples presented herein. However, those skilled in the art will understand that these aspects can be practiced without these specific details. For clarity, in some instances, the technology may be presented as comprising various functional blocks, which include devices, device components, steps, or routines in a method embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring these aspects in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the aspects.

[0099] The various aspects described above can be presented as a process or method, depicted as a flowchart, diagrammatic flowchart, data flow diagram, structural diagram, or block diagram. Although a flowchart can describe operations as a sequential process, many operations within an operation can be executed in parallel or concurrently. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but a process may have additional steps not included in the accompanying diagram. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0100] The processes and methods described in the examples above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that configure, or otherwise configure, a general-purpose computer, special-purpose computer, or processing device to perform a function or group of functions. The portion may be accessible via a network of the computer resources used. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that can be used to store the instructions, the information used, and / or information created during the methods according to the described examples include disks or optical discs, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0101] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or interlocking cards. By further example, such functionality may also be implemented on circuit boards of different chips or different processes executed on a single device.

[0102] Instructions, media for delivering such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0103] In the foregoing description, aspects of this application have been described with reference to their specific aspects, but those skilled in the art will recognize that this application is not limited thereto. Therefore, although illustrative aspects of this application have been described in detail herein, it is to be understood that the inventive concept can be implemented and employed in a variety of other ways, and the appended claims are not intended to be construed as including such variations unless limited by prior art. The various features and aspects of the applications described above may be used individually or in combination. Furthermore, aspects may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that, in alternative aspects, the methods may be performed in a different order than described.

[0104] Those skilled in the art will understand that, without departing from the scope of this description, the less than (“<”) and greater than (“>”) symbols or terms used herein may be replaced with less than or equal to (“>”) respectively. ") and greater than or equal to (" The symbol ) is used instead.

[0105] When a component is described as being “configured” to perform certain operations, such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0106] The phrase “coupled to” means any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0107] Claim language or other languages ​​that state "at least one of" and / or "one or more of" in a set indicate that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, A and B and C, or any repetition is information or data (e.g., A and A, B and B, C and C, A and A and B, etc.), or any other ordering, repetition, or combination of A, B, and C. The language "at least one of" and / or "one or more of" in a set does not limit the set to the items listed in the set. For example, the language of a claim stating "at least one of A and B" or "at least one of A or B" may refer to A, B, or A and B, and may additionally include items not listed in the set of A and B. The phrases "at least one" and "one or more" are used interchangeably herein.

[0108] Claim language or other languages ​​that state "at least one processor, at least one processor is configured to," "at least one processor is configured to," "one or more processors, one or more processors are configured to," "one or more processors are configured to," etc., indicate that one or more processors (in any combination) can perform associated operations. For example, claim language stating "at least one processor, at least one processor is configured to: X, Y, and Z" means that a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each assigned a specific subset of tasks of operations X, Y, and Z, such that the multiple processors together perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, claim language stating "at least one processor, at least one processor is configured to: X, Y, and Z" could mean that any single processor can perform only at least one subset of operations X, Y, and Z.

[0109] When referring to one or more elements that perform functions (e.g., steps of a method), one element may perform all functions, or more than one element may jointly perform these functions. When more than one element jointly performs these functions, each function does not need to be performed by every single element (e.g., different functions may be performed by different elements), and / or each function does not need to be performed by only one element as a whole (e.g., different elements may perform different sub-functions of a function). Similarly, when referring to one or more elements configured to cause another element (e.g., a device) to perform functions, one element may be configured to cause another element to perform all functions, or more than one element may be jointly configured to cause another element to perform these functions.

[0110] When referring to an entity that performs or is configured to perform functions (e.g., steps of a method) (e.g., any entity or device described herein), the entity may be configured to cause one or more elements (individually or collectively) to perform those functions. One or more components of the entity may include at least one memory, at least one processor, at least one communication interface, another component configured to perform one or more of those functions, and / or any combination thereof. When referring to an entity that performs functions, the entity may be configured to cause one component to perform all functions, or to cause more than one component to perform those functions collectively. When the entity is configured to cause more than one component to perform those functions collectively, each function does not need to be performed by every single component (e.g., different functions may be performed by different components), and / or each function does not need to be performed by only one component as a whole (e.g., different components may perform different sub-functions of a function).

[0111] The various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the examples disclosed herein can be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described in general terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0112] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or integrated circuit devices with multiple uses, including applications in wireless communication devices (mobile phones) and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as propagated signals or waves.

[0113] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or means suitable for implementing the techniques described herein.

[0114] The exemplary aspects of this disclosure include: Aspect 1. An apparatus for processing one or more frames, the apparatus comprising: one or more memories configured to store a first frame and a second frame; and one or more processors coupled to the one or more memories and configured to: estimate at least one optical flow between the first frame and the second frame; generate at least one occlusion mask based on the at least one optical flow; generate at least one weighted mask based on the at least one optical flow and the at least one occlusion mask; generate at least one inter-frame optical flow based on the at least one optical flow and the at least one weighted mask; generate at least one warped frame based on at least one of the first frame or the second frame and the at least one inter-frame optical flow; and generate an inter-frame frame between the first frame and the second frame based on the at least one warped frame.

[0115] Aspect 2. The apparatus according to aspect 1, wherein the one or more processors coupled to the one or more memories are configured to estimate the at least one optical flow between the first frame and the second frame by estimating a first optical flow in a forward direction from the first frame to the second frame; and estimating a second optical flow in a backward direction from the second frame to the first frame.

[0116] Aspect 3. The apparatus according to aspect 2, wherein the one or more processors coupled to the one or more memories are configured to generate the at least one occlusion mask by generating a first occlusion mask and a second occlusion mask based on the first optical flow and the second optical flow.

[0117] Aspect 4. The apparatus according to aspect 3, wherein forward-backward consistency is used to generate the first occlusion mask and the second occlusion mask.

[0118] Aspect 5. The apparatus according to any one of Aspects 3 or 4, wherein the one or more processors coupled to the one or more memories are configured to generate the at least one weighted mask by: generating a first weighted mask based on the first optical flow and the first occlusion mask; and generating a second weighted mask based on the second optical flow and the second occlusion mask.

[0119] Aspect 6. The apparatus according to aspect 5, wherein the one or more processors coupled to the one or more memories are configured to generate the at least one inter-frame optical flow by: generating a first inter-frame optical flow in a first direction based on the first optical flow and the first weighted mask; and generating a second inter-frame optical flow in a second direction based on the second optical flow and the second weighted mask.

[0120] Aspect 7. The apparatus according to aspect 6, wherein the one or more processors coupled to the one or more memories are configured to generate the at least one distorted frame by: generating a first distorted frame based on the first frame and the first inter-frame optical flow in the first direction; and generating a second distorted frame based on the second frame and the second inter-frame optical flow in the second direction.

[0121] Aspect 8. The apparatus according to aspect 7, wherein the one or more processors coupled to the one or more memories are configured to fuse the first distorted frame and the second distorted frame to generate the inter-frame frame.

[0122] Aspect 9. The apparatus according to any one of aspects 1 to 8, wherein the one or more processors coupled to the one or more memories are configured to: apply a neural network to the inter-frames to generate refined inter-frames.

[0123] Aspect 10. The apparatus according to aspect 9, wherein the neural network is trained on training data having one or more ground real-time inter-frames and one or more optical flow maps generated using the neural network.

[0124] Aspect 11. The apparatus according to any one of Aspects 1 to 10, wherein the at least one weighted mask includes a weight value based on an occlusion value in the at least one occlusion mask.

[0125] Aspect 12. A method for processing one or more frames, the method comprising: estimating at least one optical flow between a first frame and a second frame; generating at least one occlusion mask based on the at least one optical flow; generating at least one weighted mask based on the at least one optical flow and the at least one occlusion mask; generating at least one inter-frame optical flow based on the at least one optical flow and the at least one weighted mask; generating at least one warped frame based on at least one of the first frame or the second frame and the at least one inter-frame optical flow; and generating an inter-frame frame between the first frame and the second frame based on the at least one warped frame.

[0126] Aspect 13. The method according to aspect 12, wherein estimating the at least one optical flow between the first frame and the second frame comprises: estimating a first optical flow in a forward direction from the first frame to the second frame; and estimating a second optical flow in a backward direction from the second frame to the first frame.

[0127] Aspect 14. The method according to aspect 13, wherein generating the at least one occlusion mask: generating a first occlusion mask and a second occlusion mask based on the first optical flow and the second optical flow.

[0128] Aspect 15. The method according to aspect 14, wherein forward-backward consistency is used to generate the first occlusion mask and the second occlusion mask.

[0129] Aspect 16. The method according to any one of Aspects 14 or 15, wherein generating the at least one weighted mask comprises: generating a first weighted mask based on the first optical flow and the first occlusion mask; and generating a second weighted mask based on the second optical flow and the second occlusion mask.

[0130] Aspect 17. The method according to aspect 16, wherein generating the at least one inter-frame optical flow comprises: generating a first inter-frame optical flow in a first direction based on the first optical flow and the first weighted mask; and generating a second inter-frame optical flow in a second direction based on the second optical flow and the second weighted mask.

[0131] Aspect 18. The method according to aspect 17, wherein generating the at least one distorted frame comprises: generating a first distorted frame based on the first frame and the first inter-frame optical flow in the first direction; and generating a second distorted frame based on the second frame and the second inter-frame optical flow in the second direction.

[0132] Aspect 19. The method according to aspect 18, the method further comprising: fusing the first distorted frame and the second distorted frame to generate the inter-frame frame.

[0133] Aspect 20. The method according to any one of aspects 12 to 19, the method further comprising: applying a neural network to the inter-frame to generate a refined inter-frame.

[0134] Aspect 21. The method according to aspect 20, wherein the neural network is trained on training data having one or more ground real-time inter-frames and one or more optical flow maps generated using the neural network.

[0135] Aspect 22. The method according to any one of Aspects 12 to 21, wherein the at least one weighted mask includes weight values ​​based on occlusion values ​​in the at least one occlusion mask.

[0136] Aspect 23. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, configuring the one or more processors to: estimate at least one optical flow between a first frame and a second frame; generate at least one occlusion mask based on the at least one optical flow; generate at least one weighted mask based on the at least one optical flow and the at least one occlusion mask; generate at least one inter-frame optical flow based on the at least one optical flow and the at least one weighted mask; generate at least one distorted frame based on at least one of the first frame or the second frame and the at least one inter-frame optical flow; and generate an inter-frame frame between the first frame and the second frame based on the at least one distorted frame.

[0137] Aspect 24. The non-transitory computer-readable medium according to aspect 23, wherein the instructions, when executed by the one or more processors, cause the one or more processors to perform the operations according to any one of aspects 12 to 22.

[0138] Aspect 25. An apparatus comprising: means for estimating at least one optical flow between a first frame and a second frame; means for generating at least one occlusion mask based on the at least one optical flow; means for generating at least one weighted mask based on the at least one optical flow and the at least one occlusion mask; means for generating at least one inter-frame optical flow based on the at least one optical flow and the at least one weighted mask; means for generating at least one distorted frame based on at least one of the first frame or the second frame and the at least one inter-frame optical flow; and means for generating an inter-frame frame between the first frame and the second frame based on the at least one distorted frame.

[0139] Aspect 26. The apparatus according to aspect 25, further comprising: one or more components for performing the operation according to any one of aspects 12 to 22.

Claims

1. An apparatus for processing one or more frames, the apparatus comprising: One or more memories, the one or more memories being configured to store a first frame and a second frame; and One or more processors, said one or more processors being coupled to said one or more memories and configured to: Estimate at least one optical flow between the first frame and the second frame; At least one occlusion mask is generated based on the at least one optical flow; At least one weighted mask is generated based on the at least one optical flow and the at least one occlusion mask; At least one inter-frame optical flow is generated based on the at least one optical flow and the at least one weighted mask; At least one distorted frame is generated based on at least one of the first frame or the second frame and the at least one inter-frame optical flow; as well as An inter-frame between the first frame and the second frame is generated based on the at least one distorted frame.

2. The apparatus of claim 1, wherein the one or more processors coupled to the one or more memories are configured to estimate the at least one optical flow between the first frame and the second frame by: Estimate the first optical flow in the forward direction from the first frame to the second frame; and Estimate the second optical flow in the backward direction from the second frame to the first frame.

3. The apparatus of claim 2, wherein the one or more processors coupled to the one or more memories are configured to generate the at least one occlusion mask by: A first occlusion mask and a second occlusion mask are generated based on the first optical flow and the second optical flow.

4. The apparatus of claim 3, wherein forward-backward consistency is used to generate the first occlusion mask and the second occlusion mask.

5. The apparatus of claim 3, wherein the one or more processors coupled to the one or more memories are configured to generate the at least one weighted mask by: A first weighted mask is generated based on the first optical flow and the first occlusion mask; and A second weighted mask is generated based on the second optical flow and the second occlusion mask.

6. The apparatus of claim 5, wherein the one or more processors coupled to the one or more memories are configured to generate the at least one inter-frame optical flow by: Based on the first optical flow and the first weighted mask, a first inter-frame optical flow is generated in the first direction; and Based on the second optical flow and the second weighted mask, a second inter-frame optical flow is generated in the second direction.

7. The apparatus of claim 6, wherein the one or more processors coupled to the one or more memories are configured to generate the at least one warped frame by: A first distorted frame is generated based on the first frame and the first inter-frame optical flow in the first direction; and A second distorted frame is generated based on the second frame and the second inter-frame optical flow in the second direction.

8. The apparatus of claim 7, wherein the one or more processors coupled to the one or more memories are configured to: The first distorted frame and the second distorted frame are merged to generate the inter-frame frame.

9. The apparatus of claim 1, wherein the one or more processors coupled to the one or more memories are configured to: The neural network is applied to the inter-frames to generate refined inter-frames.

10. The apparatus of claim 9, wherein the neural network is trained on training data having one or more ground-based inter-frames and one or more optical flow maps generated using the neural network.

11. The apparatus of claim 1, wherein the at least one weighted mask comprises a weight value based on an occlusion value in the at least one occlusion mask.

12. A method for processing one or more frames, the method comprising: Estimate at least one optical flow between the first and second frames; At least one occlusion mask is generated based on the at least one optical flow; At least one weighted mask is generated based on the at least one optical flow and the at least one occlusion mask; At least one inter-frame optical flow is generated based on the at least one optical flow and the at least one weighted mask; At least one distorted frame is generated based on at least one of the first frame or the second frame and the at least one inter-frame optical flow; as well as An inter-frame between the first frame and the second frame is generated based on the at least one distorted frame.

13. The method of claim 12, wherein estimating the at least one optical flow between the first frame and the second frame comprises: Estimate the first optical flow in the forward direction from the first frame to the second frame; as well as Estimate the second optical flow in the backward direction from the second frame to the first frame.

14. The method of claim 13, wherein the at least one occlusion mask is generated: A first occlusion mask and a second occlusion mask are generated based on the first optical flow and the second optical flow.

15. The method of claim 14, wherein forward-backward consistency is used to generate the first occlusion mask and the second occlusion mask.

16. The method of claim 14, wherein generating the at least one weighted mask comprises: A first weighted mask is generated based on the first optical flow and the first occlusion mask; as well as A second weighted mask is generated based on the second optical flow and the second occlusion mask.

17. The method of claim 16, wherein generating the at least one inter-frame optical flow comprises: Based on the first optical flow and the first weighted mask, a first inter-frame optical flow is generated in the first direction; as well as Based on the second optical flow and the second weighted mask, a second inter-frame optical flow is generated in the second direction.

18. The method of claim 17, wherein generating the at least one warped frame comprises: A first distorted frame is generated based on the first frame and the first inter-frame optical flow in the first direction; as well as A second distorted frame is generated based on the second frame and the second inter-frame optical flow in the second direction.

19. The method according to claim 18, further comprising: The first distorted frame and the second distorted frame are merged to generate the inter-frame frame.

20. The method according to claim 12, further comprising: The neural network is applied to the inter-frames to generate refined inter-frames.

21. The method of claim 20, wherein the neural network is trained on training data having one or more ground-based inter-frames and one or more optical flow maps generated using the neural network.

22. The method of claim 12, wherein the at least one weighted mask comprises a weight value based on an occlusion value in the at least one occlusion mask.

23. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, configuring the one or more processors to: Estimate at least one optical flow between the first and second frames; At least one occlusion mask is generated based on the at least one optical flow; At least one weighted mask is generated based on the at least one optical flow and the at least one occlusion mask; At least one inter-frame optical flow is generated based on the at least one optical flow and the at least one weighted mask; At least one distorted frame is generated based on at least one of the first frame or the second frame and the at least one inter-frame optical flow; as well as An inter-frame between the first frame and the second frame is generated based on the at least one distorted frame.

24. An apparatus comprising: A component used to estimate at least one optical flow between the first frame and the second frame; Components for generating at least one occlusion mask based on the at least one optical flow; Components for generating at least one weighted mask based on the at least one optical flow and the at least one occlusion mask; A component for generating at least one inter-frame optical flow based on the at least one optical flow and the at least one weighted mask; Components for generating at least one distorted frame based on at least one of the first frame or the second frame and the at least one inter-frame optical flow; and A component for generating an inter-frame frame between the first frame and the second frame based on the at least one distorted frame.

Images