Segmentation mask extrapolation

Abstract

Systems and techniques for processing video data are described herein. For example, a technique may include receiving a first image after a previous image. The process may also include receiving a first segmentation mask associated with the previous image. The process may also include estimating a first set of forward motion vectors between the previous image and the first image. The process may also include estimating a reliability of the first set of forward motion vectors. The process may also include extrapolating a second segmentation mask associated with the first image using the first set of forward motion vectors and the first segmentation mask based on the estimated reliability of the first set of forward motion vectors.

Description

Technical Field

[0001] The present disclosure generally relates to processing video data.For example, aspects of the present disclosure relate to systems and techniques for generating segmentation masks (also referred to as segmentation maps) for video frames. Background Art

[0002] Many devices and systems allow video data to be processed and output for consumption. Digital video data includes a large amount of data to meet the needs of consumers and video providers. For example, consumers of video data expect high-quality video including high fidelity, resolution, frame rate, etc. Digital video data can also be used as sensor data. As an example, digital videos and images can be collected, transformed and analyzed to detect objects. One such transformation or model that can be generated includes a segmentation mask. In segmentation, pixels and objects are identified and marked with corresponding labels of specific objects (e.g., people, cars, background, etc.) within an image or video. This marking can be performed on a per-pixel basis. A segmentation mask can be a representation of the labels of an image or view.

[0003] Segmentation masks can be used in many applications and systems, including augmented reality (AR), virtual reality (VR), mixed reality (MR), robotic systems, manufacturing systems, quality assurance, automotive and aviation (e.g., manufacturing, autonomous driving or navigation, etc.), 3D scene understanding, object grasping, object tracking, video analysis, security systems, etc. For example, a segmentation mask can be associated with an image and allow each pixel of the image to be labeled according to the object category to predict the object in the view. Segmentation masks can facilitate the efficient operation of various systems. In an illustrative example, an autonomous vehicle can identify the shape and position of other vehicles based on the segmentation mask. As another illustrative example, a segmentation mask can be used to identify objects in the foreground (such as people) to better enable video conferencing software to overlay foreground objects on a synthetic background.

[0004] In some cases, generating high-quality segmentation masks may be resource intensive. Increasing video resolution and frame rate may increase this resource usage. Techniques for reducing the computational resource usage for generating segmentation masks may be desirable. Summary of the invention

[0005] The following presents a simplified summary of the invention related to one or more aspects disclosed herein. Therefore, the following summary of the invention should neither be considered as an exhaustive overview related to all conceived aspects, nor should it be considered to identify key or decisive elements related to all conceived aspects or to delineate the scope associated with any particular aspect. Accordingly, the following overview presents certain concepts related to one or more aspects of the mechanisms disclosed herein in a simplified form before the detailed description presented below.

[0006] Systems and techniques for improved video processing, such as video encoding and / or decoding, are described herein. For example, the system may attempt to generate a segmentation mask for a non-key frame based on a segmentation mask for a key frame using a relatively fast segmentation mask engine (such as by extrapolating the segmentation mask based on motion vectors). The system may be able to determine, via one or more checks, that the quality of the extrapolated segmentation mask may be reliable and then fall back to a relatively slow segmentation mask engine (such as a segmentation mask ML model).

[0007] In an illustrative example, a device for processing video data is provided. The device includes: at least one memory; and at least one processor (e.g., configured in a circuit), the at least one processor coupled to the at least one memory. The at least one processor is configured to: receive a first image after a previous image; receive a first segmentation mask associated with the previous image; estimate a first set of forward motion vectors between the previous image and the first image; estimate the reliability of the first set of forward motion vectors; and based on the estimated reliability of the first set of forward motion vectors, use the first set of forward motion vectors and the first segmentation mask to extrapolate a second segmentation mask associated with the first image.

[0008] In another example, a method of processing video data. The method may include: receiving a first image after a previous image; receiving a first segmentation mask associated with the previous image; estimating a first set of forward motion vectors between the previous image and the first image; estimating a reliability of the first set of forward motion vectors; and based on the estimated reliability of the first set of forward motion vectors, using the first set of forward motion vectors and the first segmentation mask to extrapolate a second segmentation mask associated with the first image.

[0009] As another example, a non-transitory computer-readable medium having instructions stored thereon, which, when executed by at least one processor, causes the at least one processor to: receive a first image after a previous image; receive a first segmentation mask associated with the previous image; estimate a first set of forward motion vectors between the previous image and the first image; estimate the reliability of the first set of forward motion vectors; and based on the estimated reliability of the first set of forward motion vectors, use the first set of forward motion vectors and the first segmentation mask to extrapolate a second segmentation mask associated with the first image.

[0010] In another example, an apparatus for processing video data is provided. The apparatus includes: means for receiving a first image after a previous image; means for receiving a first segmentation mask associated with the previous image; means for estimating a first set of forward motion vectors between the previous image and the first image; means for estimating the reliability of the first set of forward motion vectors; and means for extrapolating a second segmentation mask associated with the first image using the first set of forward motion vectors and the first segmentation mask based on the estimated reliability of the first set of forward motion vectors.

[0011] In some aspects, the apparatus includes a mobile device (e.g., a mobile phone or so-called "smart phone", a tablet computer, or other type of mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a television (e.g., a network-connected television), a vehicle (or a computing device of a vehicle), or other device. In some aspects, the apparatus includes at least one camera for capturing one or more images or video frames. For example, the apparatus may include a camera (e.g., an RGB camera) or multiple cameras for capturing one or more images and / or one or more videos including video frames. In some aspects, the apparatus includes a display for displaying one or more images, videos, notifications, or other displayable data. In some aspects, the apparatus includes a transmitter configured to send one or more video frames and / or grammatical data to at least one device on a transmission medium. In some aspects, the processor includes a neural processing unit (NPU), a central processing unit (CPU), a graphics processing unit (GPU), or other processing device or component.

[0012] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0013] The foregoing and other features and embodiments will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Exemplary embodiments of the present application are described in detail below with reference to the following drawings:

[0015] Figure 1 illustrates example implementations of a system on a chip (SOC) according to some examples;

[0016] Figure 2is a diagram illustrating a system including a device configured to perform segmentation mask extrapolation with fallback as part of a video processing pipeline in accordance with aspects of the present disclosure;

[0017] Figure 3 is a diagram 300 illustrating an overview of a technique for segmentation mask extrapolation with fallback in accordance with aspects of the present disclosure;

[0018] Figure 4 Segmentation mask holes according to aspects of the present disclosure are illustrated;

[0019] Figure 5A is a diagram illustrating a first hole filling process according to aspects of the present disclosure;

[0020] Figure 5B is a diagram illustrating a second void filling an uncovered portion according to aspects of the present disclosure;

[0021] Figure 6 is a diagram illustrating a data flow of a technique for segmentation mask extrapolation with fallback in accordance with aspects of the present disclosure;

[0022] Figure 7 is a diagram illustrating an alternative data flow for techniques for segmentation mask extrapolation with fallback in accordance with aspects of the present disclosure.

[0023] Figure 8 is a flow chart illustrating a technique for segmentation mask extrapolation with fallback in accordance with aspects of the present disclosure;

[0024] Fig. 9 An example computing device architecture is illustrated that illustrates an example computing device that may implement the various techniques described herein. DETAILED DESCRIPTION

[0025] Some aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes in order to provide a thorough understanding of each embodiment of the application. However, it will be apparent that each embodiment can be put into practice without these specific details. Each drawing and description are not intended to be restrictive.

[0026] The following description provides only example embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example embodiments will provide an enabling description for implementing the example embodiments to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0027] In some cases, processing digital video data may include identifying objects present in the medium before applying effects to the medium. For example, adding a virtual background to a video conference may involve identifying objects (e.g., people) in the foreground. In some cases, objects in an image may be identified by assigning a segmentation class (e.g., people, classes, car classes, background classes, etc.) to each pixel in a frame using one or more neural networks or other machine learning (ML) models and then grouping adjacent pixels that share the segmentation class to form objects (e.g., people, cars, background, etc.) of the segmentation class. This technique may be referred to as pixel-by-pixel semantic segmentation. Pixel-by-pixel labels may be referred to as segmentation masks (also referred to herein as segmentation maps).

[0028] In some cases, pixel-by-pixel semantic segmentation may involve inputting an image into an ML model, such as a convolutional neural network (CNN). The ML model may then output a segmentation mask or segmentation map for the input image. The segmentation mask may include segmentation class information for each pixel in the frame. In some cases, the segmentation mask may be configured to maintain information only for pixels corresponding to one or more classes (e.g., for pixels classified as people), thereby isolating selected classified pixels from other classified pixels (e.g., isolating person pixels from background pixels).

[0029] Processing digital video data can be computationally intensive, especially as the demand for high-quality video data continues to grow. For example, as video conferencing becomes more mainstream, consumers may expect increasingly high-quality videos with high fidelity, high resolution, frame rates, etc., as well as increasingly high-quality effects such as virtual backgrounds, animation effects, etc. However, the large amount of processing of video data required to meet such demands may be difficult to perform in real time. For example, generating segmentation masks with high-resolution and high-frame-rate video data on numerous objects in real time may be computationally intensive and / or power-consuming.

[0030] To help reduce computational complexity, a simpler segmentation ML model can be used that can be run for each image. In some cases, the simpler segmentation ML model can perform real-time segmentation for each image. However, using a simpler segmentation ML model may involve tradeoffs, such as regarding the quality and consistency of the segmentation masks generated by the simpler segmentation ML model. Alternatively, a more complex segmentation ML model may not be run in real-time (e.g., for each image). In some cases, the segmentation ML model can be used for keyframes, while optical flow can be used for mask frames for non-keyframes.

[0031] In some cases, using an ML model to generate a segmentation mask (or segmentation map) for each frame in a video can be computationally intensive, especially as the frame rate of the video (measured in frames per second (FPS)) increases. To reduce the computational load, a fully convolutional neural network can be run on a subset of video frames (referred to as keyframes), and motion vectors can be calculated for frames between keyframes (referred to as intermediate frames or non-keyframes). For example, for a keyframe (such as frame 1), an ML model can be used to generate a corresponding segmentation mask (e.g., segmentation map) for the keyframe. For a non-keyframe, such as frame 2 located between two consecutive keyframes (e.g., frame 1 and frame 5), a motion estimate can be generated for an object in the non-keyframe using the first keyframe, a non-keyframe (e.g., frame 2), and a segmentation mask from the first keyframe. A segmentation mask for a non-keyframe (e.g., frame 2) can be generated using a segmentation mask and motion estimate for a previous frame (such as keyframe 1). However, generating a segmentation mask using motion estimation can be challenging, for example, for videos containing a relatively large amount of motion. Additionally, the further a frame is in time from a keyframe, the less accurate the motion estimation, and thus the less accurate the segmentation mask may be.

[0032] Described herein are systems, apparatuses, electronic devices, methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and techniques") for extrapolating a segmentation mask for an image (e.g., an image or frame of a video). For example, the systems and techniques may provide segmentation mask extrapolation with fallback for generating a segmentation mask for an image of a video. When performing segmentation mask extrapolation with fallback, a segmentation mask ML model may be used to generate a segmentation mask (or segmentation map) for a key frame of the video. For a current non-key frame, a set of forward motion vectors may be estimated based on a previous image and a current image. The reliability of the set of forward motion vectors may be evaluated based on certain reliability criteria or heuristics. In some cases, if the set of forward motion vectors is determined to be reliable, the segmentation mask may be extrapolated using the segmentation mask of the previous image and the set of forward motion vectors. In some cases, if the set of forward motion vectors is determined to be unreliable, a fallback to the segmentation mask ML model may be triggered. In the fallback case, the segmentation mask ML model may be used to generate a segmentation mask for the current image.

[0033] Various aspects of the disclosure will be described with respect to various figures. Various aspects of the disclosure are initially described in the context of a multimedia system. Specific examples of devices and media processing systems for segmentation mask extrapolation are then described. Various aspects of the disclosure are further illustrated by and described with reference to apparatus diagrams, system diagrams, and flow diagrams relating to segmentation mask extrapolation.

[0034] Figure 1An example implementation of a system on a chip (SOC) 100 is illustrated, which may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more of the functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, task information, and other information may be stored in a memory block associated with a neural processing unit (NPU) 108, in a memory block associated with the CPU 102, in a memory block associated with a graphics processing unit (GPU) 104, in a memory block associated with a digital signal processor (DSP) 106, in a memory block 118, and / or may be distributed among multiple blocks. Instructions executed at the CPU 102 may be loaded from a program memory associated with the CPU 102, or may be loaded from the memory block 118.

[0035] The SOC 100 may also include additional processing blocks tailored for specific functions, such as a GPU 104, a DSP 106, a connectivity block 110 (which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and a multimedia processor 112 that may, for example, detect and recognize gestures. In one specific implementation, the NPU is implemented in the CPU 102, the DSP 106, and / or the GPU 104. The SOC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or a navigation module 120, which may include a global positioning system.

[0036] The SOC 100 may be based on the ARM instruction set. The SOC 100 and / or its components may be configured to perform segmentation mask extrapolation. For example, the CPU 102, the DSP 106, and / or the GPU 104 may be configured to perform segmentation mask extrapolation by instructions included in a software application program at least partially stored on the memory 118.

[0037] Figure 2A system 200 is depicted that includes a device 202 configured to perform segmentation mask extrapolation with fallback as part of a video processing pipeline 210. The device 202 is coupled to a camera 207 and a storage medium 214 (e.g., a data storage device). In some implementations, the camera 207 is configured to provide image data 208 (e.g., a video data stream) to a processor 204 for processing by the video processing pipeline 210. The processor 204 can be a CPU, a DSP, a GPU, or other processor capable of processing video data. In some implementations, the device 202 can be coupled to and / or can include multiple cameras (e.g., a dual camera system, three cameras, or other number of cameras). In some cases, the device 202 can be coupled to a microphone and / or other input device (e.g., a keyboard, a mouse, a touch input device (such as a touch screen and / or a touchpad), and / or other input devices). In some examples, the camera 207, the storage medium 214, the microphone, and / or other input devices can be part of the device 202.

[0038] The device 202 is also coupled to the second device 290 via a transmission medium 218 such as one or more wireless networks, one or more wired networks, or a combination thereof. For example, the transmission medium 218 may include a channel provided by a wireless network, a wired network, or a combination of a wired network and a wireless network. The transmission medium 218 may form a portion of a packet-based network such as a local area network, a wide area network, or a global network (such as the Internet). The transmission medium 218 may include a router, a switch, a base station, or any other equipment that may be useful for facilitating communication from a source device to a receiving device. The wireless network may include any wireless interface or a combination of wireless interfaces, and may include any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, WiFiTM, radio frequency (RF), UWB, WiFi direct connection, cellular, long-term evolution (LTE), WiMaxTM, etc.). The wired network may include any wired interface (e.g., optical fiber, Ethernet, power line Ethernet, Ethernet based on coaxial cable, digital signal line (DSL), etc.). The wired network and / or wireless network may be implemented using various equipment (such as base stations, routers, access points, bridges, gateways, switches, etc.). The encoded video bitstream data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to a receiving device.In some cases, device 290 may be substantially similar to device 202.

[0039] The device 202 includes one or more processors (referred to herein as "processors") 204 coupled to a memory 206, a first interface ("I / F 1") 212, and a second interface ("I / F 2") 216. The processor 204 is configured to receive image data 208 from a camera 207, from the memory 206, and / or from a storage medium 214. The processor 204 is coupled to the storage medium 214 via the first interface 212 (e.g., via a memory bus), and to the transmission medium 218 via the second interface 216 (e.g., a network interface device, a wireless transceiver and antenna, one or more other network interface devices, or a combination thereof).

[0040] The processor 204 includes a video processing pipeline 210. Input data 270 may be input to the video processing pipeline 210. The input data 270 may be one or more images of a video. In some cases, the input data 270 may be obtained by the camera 207. The video processing pipeline 210 includes a segmentation engine 260. The segmentation engine 260 is configured to receive the input data 270 and process the input data 270 to generate segmentation data (such as a segmentation mask) based on the input data 270. The segmentation data may be output from the segmentation engine 260 to one or more additional processing engines 262 for further processing. After being processed by the one or more additional processing engines 262, output data 272 may be generated. In some cases, the output data 272 may be stored in the storage medium 214, for example, via the I / F 1 212. In some examples, the output data 272 may be sent to the device 290 via the transmission medium 218 and the I / F 2 216.

[0041] As indicated above, some implementations of processor 204 include multiple processors and elements of video processing pipeline 210 may be included on (e.g., executed by) different ones of the multiple processors. For example, segmentation engine 260 may be included on a custom processor such as an application specific integrated circuit (ASIC).

[0042] Figure 3 is a diagram 300 illustrating an overview of a technique for segmentation mask extrapolation with fallback in accordance with aspects of the present disclosure. In diagram 300, a current image 302 captured at time t may be input to a motion vector estimator 306 along with a previous image 304 captured at time t-1. The current image 302 is checked at step 308 to see if the current image 302 is a key frame.

[0043] If the current image 302 is a keyframe, the current image 302 may be passed to a segmentation mask ML model 310. In some cases, the segmentation mask ML model 310 may be a relatively high quality segmentation mask ML model. In some cases, a relatively high quality segmentation mask ML model may not be able to execute in real time on a system or device on which the ML model is being executed. In some cases, a relatively high quality segmentation mask ML model may be able to execute in real time, but may consume a relatively large amount of power, for example, compared to segmentation mask extrapolation. The segmentation mask ML model 310 may generate a segmentation mask 312 for the current image 302 for output. As shown, the segmentation mask 312 may label and group pixels (such as for people and buildings in the current image 302) according to categories for the current image 302. It should be understood that the segmentation mask 312 is shown for clarity, and the data represented by the segmentation mask 312 may be associated with the input image (e.g., attached to the input image, added to the input image, etc.), for example, as metadata.

[0044] In some cases, the segmentation mask 312 may be input to a threshold mask engine 314. The threshold mask engine 314 may apply a threshold mask on the segmentation mask 312 to mark portions of the segmentation mask 312 as being part of the foreground or background. In some cases, the threshold mask engine 314 may mark individual pixels. In some cases, the threshold mask engine 314 may mask out pixels of uninteresting objects so that the uninteresting objects may be merged as part of the background. For example, a threshold mask engine 314 of a video conferencing application may mask out chairs, windows, etc. in the background (e.g., merge chairs, windows, etc. into the background). In other cases, the threshold mask engine 314 may mark portions of the threshold mask by block. For example, the threshold mask engine may mark an 8×8 pixel block as a member of the foreground or background. For example, block-based masking may be used, where the motion vector estimator engine 306 generates block-based optical flow information between the current image 302 and the previous image 304.

[0045] In diagram 300, the representation of the threshold segmentation mask 316 illustrates objects marked as foreground objects and omits objects marked as background objects. It should be understood that the representation of the threshold segmentation mask 316 is shown for clarity and that the data of the representation of the threshold segmentation mask 316 may include indications for both background objects and foreground objects. The threshold segmentation mask may be input to an extrapolation / backoff pipeline 318.

[0046] If it is determined at step 308 that the current image 302 is not a key frame, an indication may be passed to a motion vector estimator engine 306. The motion vector estimator engine 306 may estimate a motion vector for the current image 302. A motion vector describes the difference between one image and another (e.g., optical flow). In some cases, the motion vector estimator engine 306 may generate motion vectors between consecutive frames (such as the current image 302 and the previous image 304). In some cases, the motion vector estimator engine 306 may generate motion vectors in a forward direction (e.g., forward in time, such as from time t-1 to time t). In some cases, a backward motion vector may also be generated, for example, to check the reliability of the forward motion vector. In some cases, motion vectors may be generated in a manner similar to motion estimation for video encoding / decoding. Motion vectors may be generated on a per-pixel or block level.

[0047] The generated motion vectors and the current image 302 may be passed to an image hole filling engine 320 of the extrapolation / backoff pipeline 318. As an object moves, areas behind the object may become visible. If these uncovered areas were not visible in the previous image, there may not be any motion vectors in the vector field (e.g., holes in the vector field) to describe the uncovered areas. The image hole filling engine 320 detects and fills these holes in the vector field. The filled motion vectors and the current image 302 may then be input to a backoff heuristic engine 322. The operations performed by the image hole filling engine 320 are discussed in more detail below.

[0048] The fallback heuristic engine 322 may measure the set of heuristics to determine whether the extrapolated segmentation mask is expected to be inaccurate based on, for example, motion vectors from the current image 302. If the extrapolated segmentation mask is expected to be inaccurate, the extrapolation / fallback pipeline 318 may reject the extrapolated segmentation mask and instead run the segmentation mask ML model 310 on the current image 302. If the extrapolated segmentation mask is not expected to be inaccurate, the extrapolation / fallback pipeline 318 may continue to use the extrapolation engine 326 to extrapolate the segmentation mask.

[0049] In some cases, if there is too much foreground motion, the extrapolated segmentation mask may not be accurate. To help detect this foreground motion, the fallback heuristic engine 322 may determine one or more statistics based on the set of motion vectors and apply certain heuristic checks. As a first example heuristic check, the fallback heuristic engine 322 may determine the gradient of the foreground motion vector using the threshold segmentation mask 316 from the previous image 304 and the populated motion vectors for the current image 302. At step 324, the determined gradient may be compared to a gradient threshold. If the gradient is above the gradient threshold, a fallback to the segmentation mask ML model 310 may be triggered at step 324. The extrapolation / fallback pipeline 318 may pass the current image to the segmentation mask ML model 310, and the segmentation mask ML model 310 may be run on the current image 302.

[0050] As a second example heuristic check, the fallback heuristic engine 322 can determine whether to fall back to the segmentation mask ML model 310 based on the percentage change of the foreground objects and their corresponding motion vectors. For example, based on the threshold segmentation mask 316 from the previous image 304 and the populated motion vectors for the current image 302, the fallback heuristic engine 322 can determine whether more than a threshold percentage (e.g., value) of foreground pixels have moved. If more than a threshold percentage of foreground pixels have moved, then a fallback to the segmentation mask ML model 310 can be triggered at step 324. The extrapolation / fallback pipeline 318 can pass the current image to the segmentation mask ML model 310, and the segmentation mask ML model 310 can be run on the current image 302.

[0051] As a third example heuristic check, the fallback heuristic engine 322 can determine the reliability of the foreground motion vector. The fallback heuristic engine 322 can determine the reliability of the foreground motion vector in a variety of ways, such as by comparing the forward motion vector and the backward motion vector to see how consistent the forward motion vector and the backward motion vector are (e.g., the amount of difference between the negative of the forward motion vector and the backward motion vector), how spatially relevant the motion vectors are for an object, the input and output spatial image distribution, or any other suitable reliability measure. If the fallback heuristic engine 322 determines that the reliability is low, then a fallback to the segmentation mask ML model 310 can be triggered at step 324. The extrapolation / fallback pipeline 318 can pass the current image to the segmentation mask ML model 310, and the segmentation mask ML model 310 can be run on the current image 302.

[0052] If the set of heuristics indicates at step 324 that the extrapolated segmentation mask is not expected to be inaccurate, the previous mask 316 and the populated motion vectors for the current image 302 may be passed from the fallback heuristic engine 322 to the segmentation mask extrapolation engine 326. The thresholded segmentation mask 316 from the previous image 304 may also be input to the segmentation mask extrapolation engine 326. The segmentation mask extrapolation engine 326 may interpolate from the thresholded segmentation mask 316 from the previous image 304 based on the populated motion vectors for the current image 302 to generate a segmentation mask 312 for the current image 302.

[0053] Generating the segmentation mask 312 by extrapolating from a previous segmentation mask when conditions are favorable allows for a more accurate and higher quality segmentation mask to be generated by extrapolation. Similarly, falling back to the segmentation mask ML model 310 when conditions are not favorable for extrapolating from a previous segmentation mask allows for appropriate use of the segmentation mask ML model 310 by balancing the power and resource savings of using extrapolation instead of the segmentation mask ML model 310 while allowing for increased accuracy compared to using extrapolation alone.

[0054] Figure 4 Segmentation mask holes according to aspects of the present disclosure are illustrated. In some cases, each pixel of a segmentation mask, such as segmentation mask 402, may be labeled. In some cases, pixels may be grouped into blocks and foreground / background labeling may be performed on a block-by-block basis rather than per pixel. Segmentation mask 402 shows two labels for the pixels of the segmentation mask, namely background object 404 and foreground object 406. For pixels associated with foreground object 406, segmentation mask 402 may be associated with a motion vector of 8 pixels down. In segmentation mask 450, the motion vector is applied and the pixels of foreground object 406 are moved down by 8 pixels.

[0055] In some cases, it can be assumed that when a foreground object moves, portions of the background become exposed. In this example, the movement of the pixels of the foreground object 406 exposes a portion 452 (e.g., a hole) of the background object 404 that is not visible in the segmentation mask 402. Since the portion is not visible in the segmentation mask 402, there may be no image information or motion vectors associated with the pixels of the portion 452. In some cases, a hole filling process can be used to generate image information and motion vectors for the pixels of the portion 452 based on neighboring pixels of the background.

[0056] Figure 5A5 is a diagram illustrating a first hole filling process according to aspects of the present disclosure. Diagram 500 includes rows of blocks of images of a current image 502A and a previous image 504A before filling a first hole 506A and rows of pixels of a current image 502B and a previous image 504B after filling a first hole 506B. A set of motion vectors 508 illustrates where blocks of the previous image 504A have been moved to in the current image 502A.

[0057] In diagram 500, foreground block 510 is associated with a foreground object (e.g., labeled as a foreground object), and background block 512 is associated with a background object (e.g., labeled as a background object). In previous image 504A, foreground block 510 was adjacent to background block 512. As depicted by motion vector 508, foreground block 510 has moved to the left in current image 502A compared to previous image 504A, while background block 512 has moved to the right, leaving uncovered portion 514 (e.g., a hole). In this example, uncovered portion 514 is three blocks in size, and first hole 506A corresponds to the first block of uncovered portion 514.

[0058] In some cases, the uncovered portion may be detected based on blocks in the current image 502A that are not associated with one or more motion vectors from the previous image. After finding the uncovered portion, the uncovered portion may be filled in. In some cases, it may be assumed that when a foreground object (such as the foreground object associated with the foreground block 510) is present, portions of the background (such as the background associated with the background block 512) may not be covered.

[0059] Under the assumption that the foreground covers the background, the adjacent background blocks of the motion vector and the corresponding pixel information may be used to fill the uncovered blocks (such as the uncovered portion 514). In some cases, filling the uncovered blocks with background blocks may be performed by searching for the closest (e.g., nearest) foreground block and copying the motion vector of the foreground block to the uncovered block. For example, for the first hole 506B of the uncovered portion 514, the search may indicate that the closest foreground block 510 is the block 518 in the current image 502B. The motion vector 516 corresponding to the block 522 describing the motion from the block 518 of the previous image 502B to the current image 504B may be copied as the motion vector 520, and the copied motion vector 520 may be assigned to the first hole 506B.

[0060] Figure 5B550 is a diagram illustrating filling the second hole 556 of the uncovered portion 514 according to aspects of the present disclosure. In some cases, the motion vector of the closest foreground block may be used multiple times. For example, the foreground block 518 may be the closest foreground block to the second hole 556 of the uncovered portion 514. In this case, the motion vector 516 associated with the closest foreground block 518 is copied as the motion vector 558, and the copied motion vector 558 is assigned to the second hole 556.

[0061] Figure 6 is a diagram 600 illustrating a flow diagram of a technique for segmentation mask extrapolation according to aspects of the present disclosure. In diagram 600, input image (0) may be a key frame for an input video input at t=0. Input image (0) may be passed directly to a segmentation mask ML model (0) step at t=2 to generate an output image (0) with a corresponding segmentation mask at t=3. In this example, the segmentation mask ML model may be able to generate the segmentation mask in real time (e.g., in a time frame less than or equal to the frame rate of the video being processed).

[0062] The next input image may be received at the input image (1) step at t=1, and the next input image may not be a key frame. Therefore, the next input image may be passed to the motion estimation (1) step, which may be performed by a motion vector estimation engine (such as the motion vector estimator engine 306). The input image from step input image (0) may also be used for motion vector estimation in the motion estimation (1) step. The output of the motion estimation (1) step may be passed to the MV processing (1) step, which may be performed by the extrapolation / backoff pipeline 318 for motion vector processing.

[0063] Motion vector processing may include uncovered portion filling and heuristic determination and evaluation, such as by Figure 3 The processing performed by the image hole filling engine 320, the fallback heuristic engine 322 and the extrapolation quality check step 324. Based on the determination that the motion vector may be reliable at the MV processing (1) step, the extrapolation (1) step performed by the segmentation mask extrapolation engine 326, for example, extrapolates the segmentation mask based on the motion vector estimated at t=3, and outputs the image and the corresponding segmentation mask at the output image (1) step at t=4.

[0064] Another input image may be received at the input image (2) step at t=2. At the motion estimation (2) step, a motion vector between the other image and the next input image received at the input image step (1) may be estimated at the motion estimation (2) step. The output of the motion estimation (2) step may be passed to the MV processing (2) step. In this example, the MV processing (2) step may determine that the motion vector may be unreliable. Based on the determination that the motion vector may be unreliable, the other input image received at the input image (2) step may be passed to the segmentation mask ML model for processing at the segmentation mask ML (2) step. The segmentation mask and corresponding image may be output at step output image (2) at t=5.

[0065] In diagram 600, the image input at step Input Image (3) at t = 3 may have a segmentation mask generated via extrapolation in a similar manner as the image input at t = 1. The image input at step Input Image (4) at t = 4 may be a keyframe and may be processed by the Segmentation Mask ML model at the Segmentation Mask ML (4) step.

[0066] In cases where the segmentation mask ML model can run in real time, the use of the segmentation mask ML model can be reduced by extrapolating the segmentation mask and falling back to the segmentation mask ML model as needed, as shown in diagram 600. This can help trade off the use of relatively resource-intensive ML model processing with relatively less resource-intensive motion vector generation / processing and segmentation mask extrapolation.

[0067] In some cases, the segmentation mask ML model cannot execute in real time (e.g., cannot execute in less time than the frame rate of the video being processed) for the data flow. Figure 7 700 is a diagram illustrating a data flow for a technique for segmentation mask extrapolation with fallback according to aspects of the present disclosure. The data flow illustrated in diagram 700 is similar to the data flow illustrated in diagram 600. The main difference is that the execution time of the segmentation mask ML model can span multiple time periods N. In diagram 700, the execution time of the segmentation mask ML model can span 3 time periods (e.g., N=3). For example, a key frame image input at the input image (0) step can be passed to the segmentation mask ML (0) step at t=1. The segmentation mask ML step (0) can be completed after t=3 and the segmentation mask and image can be output at the output image (0) step at t=4.

[0068] A second input image that is not a keyframe may be received at the input image (1) step at t=1. The second input image may be passed to the motion estimation (1) step. The input image from step input image (0) may also be used for motion vector estimation and may also be used in the motion estimation (1) step. The output of the motion estimation (1) step may be passed to the MV processing (1) step. Based on determining that the motion vectors may be reliable at the MV processing (1) step, an extrapolation (1) step may be performed at t=4 to extrapolate a segmentation mask for the second input image at t=5.

[0069] A third input image that is not a keyframe may be received at the input image (2) step at t=2. The third input image may be passed to the motion estimation (2) step. The input image from step input image (1) may also be used for motion vector estimation and may also be used in the motion estimation (2) step. The output of the motion estimation (2) step may be passed to the MV processing (2) step. Based on a determination that the motion vector estimated in the motion estimation (2) step is unreliable, the third input image may be passed to the segmentation mask ML model at the segmentation mask ML (2) step. The segmentation mask and corresponding image may be output at step output image (2) at t=6.

[0070] A fourth input image that is also not a keyframe may be received at the input image (3) step at t=3. Motion estimation may be performed on the fourth input image at the motion estimation (3) step. In some cases, since the execution time of the segmentation mask ML model may span multiple time periods, a time delay may be inserted to help sort the output images. In diagram 700, the output from the motion estimation (3) step may be ready at t=4, but is not acted upon by the MV processing (3) step until t=6. Motion vector processing and extrapolation may be performed in a manner consistent with that discussed above with respect to the second input image and the MV processing (1) step and the extrapolation (1) step. In some cases, fallback may not be triggered at a rate higher than 1 / N, or in the above example, in one of the three images. As shown in diagram 700, triggering fallback at a rate higher than 1 / N may result in a delayed output image. In some cases, a wait time may be added to account for the amount of time used by the segmentation mask ML model.

[0071] In some cases, multiple segmentation ML models may be used in conjunction with segmentation mask extrapolation with fallback techniques. For example, a relatively more complex, more accurate segmentation mask ML model may be used for key frames. Such a relatively more complex segmentation mask ML model may not be able to run at the frame rate of the input video. For non-key frames, a relatively simpler, less accurate segmentation mask ML model that can run at the frame rate of the input video may be used to fall back from the segmentation mask extrapolation.

[0072] Figure 8800 is a flowchart illustrating a process 800 for processing video data (e.g., to generate one or more segmentation masks) according to various aspects of the present disclosure. Process 800 may be performed by a computing device (or apparatus) or a component of a computing device (e.g., a chipset, a codec, etc.). The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable device such as a watch, an extended reality (XR) device such as a virtual reality (VR) device or an augmented reality (AR) device, a vehicle or a component or system of a vehicle, or other type of computing device. In some cases, the computing device may be or may include a decoding device, such as an encoding device 104, a decoding device 112, or a combined encoding device (or codec). The operations of process 800 may be implemented as a software component that executes and runs on one or more processors.

[0073] At box 802, a computing device (or a component thereof) may receive a first image after a previous image. At box 804, a computing device (or a component thereof) may receive a first segmentation mask associated with the previous image. In some cases, the first image is a non-key frame, and the segmentation mask machine learning model is configured to generate a segmentation mask for a key frame. At box 806, a computing device (or a component thereof) may estimate a first set of forward motion vectors between the previous image and the first image. In some cases, the computing device (or a component thereof) may mark one or more portions of the first segmentation mask as part of a foreground or background. In some cases, the computing device (or a component thereof) may: detect an uncovered portion of the first image based on the first set of forward motion vectors; and fill the uncovered portion based on adjacent background portions of the first segmentation mask and the previous image. In some cases, filling the uncovered portion may include: identifying a motion vector associated with a foreground portion of the first segmentation mask that is closest to a portion of the uncovered portion; copying the motion vector; and associating the copied motion vector with the portion of the uncovered portion.

[0074] At block 808, the computing device (or a component thereof) may estimate the reliability of the first set of forward motion vectors. In some cases, estimating the reliability of the first set of forward motion vectors includes: determining one or more heuristic statistics based on the first set of forward motion vectors; and evaluating the one or more heuristic statistics to estimate the reliability of the first set of forward motion vectors. In some cases, the one or more heuristic statistics include at least one of: a gradient for one or more motion vectors of the first set of forward motion vectors, or a percentage change in foreground. In some cases, evaluating the one or more heuristic statistics includes at least one of: evaluating the gradient for one or more motion vectors against a gradient threshold, or estimating the percentage change in foreground based on a threshold for foreground motion vectors.

[0075] At block 810, the computing device (or a component thereof) may use the first set of forward motion vectors and the first segmentation mask to extrapolate a second segmentation mask associated with the first image based on the estimated reliability of the first set of forward motion vectors. In some cases, the computing device (or a component thereof) may receive a second image, wherein the second image is a non-keyframe, wherein the segmentation mask machine learning model is configured to generate a segmentation mask for a keyframe; estimate a second set of forward motion vectors between the second image and another previous image; estimate the reliability of the second set of forward motion vectors; and determine to apply the segmentation mask machine learning model to the second image based on estimating that the second set of forward motion vectors is unreliable. In some cases, the computing device (or a component thereof) may receive a third image, wherein the third image is a keyframe, wherein the segmentation mask machine learning model is configured to generate a segmentation mask for the keyframe; and apply the segmentation mask machine learning model to the third image.

[0076] In some examples, the techniques or processes described herein may be performed by a computing device or apparatus (such as encoding device 104, decoding device 112, and / or any other computing device). In some cases, a computing device or apparatus may include a processor, a microprocessor, a microcomputer, or other components of a device configured to perform the steps of the process described herein. In some examples, a computing device or apparatus may include a camera configured to capture video data (e.g., a video sequence) including video frames. For example, a computing device may include a camera device, which may or may not include a video codec. As another example, a computing device may include a mobile device with a camera (e.g., a camera device such as a digital camera, an IP camera, etc., a mobile phone or tablet computer including a camera, or other types of devices with a camera). In some cases, a computing device may include a display for displaying an image. In some examples, a camera or other capture device that captures video data is separated from a computing device, in which case the computing device receives the captured video data. The computing device may also include a network interface, a transceiver, and / or a transmitter configured to transmit video data. The network interface, the transceiver, and / or the transmitter may be configured to transmit data based on an Internet Protocol (IP) or other network data.

[0077] The processes described herein may be implemented in hardware, computer instructions, or any combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0078] Additionally, the processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, implemented by hardware, or a combination of the above. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0079] Fig. 9 An example computing device architecture 900 is illustrated of an example computing device that can implement the various techniques described herein. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device of a vehicle), or other device. For example, the computing device architecture 900 may be used as Figure 2 Components of the computing device architecture 900 are shown as being in electrical communication with each other using a connection 905, such as a bus. The example computing device architecture 900 includes a processing unit (CPU or processor) 910 and a computing device connection 905 that couples various computing device components including computing device memory 915, such as a read-only memory (ROM) 920 and a random access memory (RAM) 925, to the processor 910.

[0080] The computing device architecture 900 may include a cache of high-speed memory directly connected to the processor 910, very close to the processor, or integrated as part of the processor. The computing device architecture 900 may copy data from the memory 915 and / or the storage device 930 to the cache 912 for fast access by the processor 910. In this way, the cache can provide a performance improvement that avoids the processor 910 from delaying when waiting for data. These modules and other modules can control or be configured to control the processor 910 to perform various actions. Other computing device memories 915 can also be used. The memory 915 may include a variety of different types of memory with different performance characteristics. The processor 910 may include any general-purpose processor and hardware or software services (such as service 1 932, service 2 934, and service 3 936 stored in the storage device 930) configured to control the processor 910, as well as a dedicated processor in which software instructions are incorporated into the processor design. The processor 910 may be a self-contained system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0081] To enable user interaction with the computing device architecture 900, the input device 945 may represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, etc. The output device 935 may also be one or more of a plurality of output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some cases, a multi-mode computing device may enable a user to provide multiple types of input to communicate with the computing device architecture 900. The communication interface 940 may generally command and manage user input and computing device output. There is no restriction on operating on any particular hardware arrangement, and therefore the underlying features herein may be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0082] The storage device 930 is a non-volatile memory and may be a hard disk or other type of computer readable medium that can store computer accessible data, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette tape, a random access memory (RAM) 925, a read-only memory (ROM) 920, and hybrids thereof. The storage device 930 may include services 932, 934, 936 for controlling the processor 910. Other hardware or software modules are contemplated. The storage device 930 may be connected to the computing device connector 905. In one aspect, a hardware module that performs a particular function may include a software component stored in a computer readable medium connected to the necessary hardware components (such as the processor 910, the connector 905, the output device 935, etc.) to perform the function.

[0083] Aspects of the present disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop, vehicle, drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to a particular device.

[0084] The term "device" is not limited to one or a specific number of physical objects (such as a smart phone, a controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that can implement at least some parts of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. Additionally, the term "system" is not limited to multiple components or specific embodiments. For example, a system can be implemented on one or more printed circuit boards or other substrates and can have movable or static components. Although the following description and examples use the term "system" to describe various aspects of the present disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0085] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be appreciated by those skilled in the art that embodiments may be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in the method embodied in a combination of software or hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid obscuring these embodiments in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques may be shown without necessary details to avoid confusing each embodiment.

[0086] Individual embodiments may be described above as processes or methods depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe an operation as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or main function.

[0087] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from a computer readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language), firmware, source code, etc.

[0088] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as flash memory), memory or memory devices, disks or optical disks, flash memory, USB devices provided with non-volatile memory, network storage devices, compact discs (CDs) or digital versatile discs (DVDs), any suitable combination thereof, etc. Computer-readable media may store thereon code and / or machine-executable instructions that may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0089] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals including bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0090] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0091] The instructions, the media for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0092] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary embodiments of the present application have been described in detail herein, it is to be understood that the inventive concept can be embodied and adopted in various other ways, and the appended claims are intended to be interpreted as including such variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, without departing from the broader essence and scope of this specification, the embodiments can be used in any number of environments and applications beyond the environments and applications described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that in an alternative embodiment, each method can be performed in a different order than described.

[0093] It should be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0094] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0095] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0096] Claim language or other language stating "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0097] Various exemplary logic blocks, modules, circuits and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, frames, modules, circuits and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed to the entire system. The technician can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as making it depart from the scope of the application.

[0098] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices, mobile phones, and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions that execute one or more of the above methods when executed. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0099] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein.

[0100] Exemplary aspects of the present disclosure include:

[0101] Aspect 1: A method of processing video data. The method includes: receiving a first image after a previous image; receiving a first segmentation mask associated with the previous image; estimating a first set of forward motion vectors between the previous image and the first image; estimating the reliability of the first set of forward motion vectors; and based on the estimated reliability of the first set of forward motion vectors, using the first set of forward motion vectors and the first segmentation mask to extrapolate a second segmentation mask associated with the first image.

[0102] Aspect 2: The method according to claim 1, wherein estimating the reliability of the first set of forward motion vectors comprises: determining one or more heuristic statistics based on the first set of forward motion vectors; and evaluating the one or more heuristic statistics to estimate the reliability of the first set of forward motion vectors.

[0103] Aspect 3: The method of claim 2, wherein the one or more heuristic statistics include at least one of: a gradient of one or more motion vectors of the first set of forward motion vectors, or a percentage change of foreground.

[0104] Aspect 4: The method of claim 3, wherein evaluating the one or more heuristic statistics comprises at least one of: evaluating the gradient for one or more motion vectors against a gradient threshold, or estimating the percentage change of the foreground based on a threshold of the foreground motion vector.

[0105] Aspect 5: According to the method described in any one of claims 1 to 4, the method further includes: receiving a second image, wherein the second image is a non-key frame, wherein the segmentation mask machine learning model is configured to generate a segmentation mask for the key frame; estimating a second set of forward motion vectors between the second image and another previous image; estimating the reliability of the second set of forward motion vectors; and determining to apply the segmentation mask machine learning model to the second image based on the estimation that the second set of forward motion vectors is unreliable.

[0106] Aspect 6: The method according to any one of claims 1 to 5, further comprising marking one or more parts of the first segmentation mask as part of the foreground or background.

[0107] Aspect 7: The method according to claim 6 further includes: detecting an uncovered portion of the first image based on the first set of forward motion vectors; and filling the uncovered portion based on an adjacent background portion of the first segmentation mask and the previous image.

[0108] Aspect 8: A method according to claim 7, wherein filling the uncovered portion includes: identifying a motion vector associated with a foreground portion of the first segmentation mask that is closest to a portion of the uncovered portion; copying the motion vector; and associating the copied motion vector with the portion of the uncovered portion.

[0109] Method 9: The method according to any one of claims 1 to 8, wherein the first image is a non-keyframe, and wherein the segmentation mask machine learning model is configured to generate a segmentation mask for a keyframe.

[0110] Aspect 10: According to any one of claims 1 to 9, the method further includes: receiving a third image, wherein the third image is a key frame, wherein the segmentation mask machine learning model is configured to generate a segmentation mask for the key frame; and applying the segmentation mask machine learning model to the third image.

[0111] Aspect 11: A device for processing video data, the device comprising: at least one memory; and at least one processor, the at least one processor coupled to the at least one memory, the at least one processor being configured to: receive a first image after a previous image; receive a first segmentation mask associated with the previous image; estimate a first set of forward motion vectors between the previous image and the first image; estimate the reliability of the first set of forward motion vectors; and based on the estimated reliability of the first set of forward motion vectors, use the first set of forward motion vectors and the first segmentation mask to extrapolate a second segmentation mask associated with the first image.

[0112] Aspect 12: The apparatus according to claim 11, wherein, in order to estimate the reliability of the first set of forward motion vectors, the at least one processor is further configured to: determine one or more heuristic statistics based on the first set of forward motion vectors; and evaluate the one or more heuristic statistics to estimate the reliability of the first set of forward motion vectors.

[0113] Aspect 13: The apparatus of claim 12, wherein the one or more heuristic statistics include at least one of: a gradient of one or more motion vectors of the first set of forward motion vectors, or a percentage change of foreground.

[0114] Aspect 14: The apparatus of claim 13, wherein, in order to evaluate the one or more heuristic statistics, the at least one processor is further configured to: evaluate the gradient for one or more motion vectors against a gradient threshold, or estimate the percentage change of the foreground based on a threshold of the foreground motion vector.

[0115] Aspect 15: An apparatus according to any one of claims 11 to 14, wherein the at least one processor is further configured to: receive a second image, wherein the second image is a non-key frame, wherein the segmentation mask machine learning model is configured to generate a segmentation mask for the key frame; estimate a second set of forward motion vectors between the second image and another previous image; estimate the reliability of the second set of forward motion vectors; and determine to apply the segmentation mask machine learning model to the second image based on the estimation that the second set of forward motion vectors is unreliable.

[0116] Aspect 16: The apparatus according to any one of claims 11 to 15, wherein the at least one processor is further configured to mark one or more portions of the first segmentation mask as part of the foreground or background.

[0117] Aspect 17: An apparatus according to claim 16, wherein the at least one processor is further configured to: detect an uncovered portion of the first image based on the first set of forward motion vectors; and fill the uncovered portion based on an adjacent background portion of the first segmentation mask and the previous image.

[0118] Aspect 18: An apparatus according to claim 17, wherein, in order to fill the uncovered portion, the at least one processor is further configured to: identify a motion vector associated with a foreground portion of the first segmentation mask that is closest to a portion of the uncovered portion; copy the motion vector; and associate the copied motion vector with the portion of the uncovered portion.

[0119] Aspect 19: An apparatus according to any one of claims 11 or 18, wherein the first image is a non-key frame, and wherein the segmentation mask machine learning model is configured to generate a segmentation mask for a key frame.

[0120] Aspect 20: An apparatus according to any one of claims 11 to 19, wherein the at least one processor is further configured to: receive a third image, wherein the third image is a key frame, wherein the segmentation mask machine learning model is configured to generate a segmentation mask for the key frame; and apply the segmentation mask machine learning model to the third image.

[0121] Aspect 21: A non-transitory computer-readable medium having instructions stored thereon, which, when executed by at least one processor, causes the at least one processor to: receive a first image after a previous image; receive a first segmentation mask associated with the previous image; estimate a first set of forward motion vectors between the previous image and the first image; estimate the reliability of the first set of forward motion vectors; and based on the estimated reliability of the first set of forward motion vectors, use the first set of forward motion vectors and the first segmentation mask to extrapolate a second segmentation mask associated with the first image.

[0122] Aspect 22: The non-transitory computer-readable medium of claim 21, wherein, in order to estimate the reliability of the first set of forward motion vectors, the instructions further cause the at least one processor to: determine one or more heuristic statistics based on the first set of forward motion vectors; and evaluate the one or more heuristic statistics to estimate the reliability of the first set of forward motion vectors.

[0123] Aspect 23: The non-transitory computer-readable medium of claim 22, wherein the one or more heuristic statistics include at least one of: a gradient of one or more motion vectors of the first set of forward motion vectors, or a percentage change in foreground.

[0124] Aspect 24: A non-transitory computer-readable medium according to claim 23, wherein, in order to evaluate the one or more heuristic statistics, the instructions further cause the at least one processor to: evaluate the gradient for one or more motion vectors against a gradient threshold, or estimate the percentage change of the foreground based on a threshold of the foreground motion vector.

[0125] Aspect 25: A non-transitory computer-readable medium according to any one of claims 21 to 24, wherein the instructions further cause the at least one processor to: receive a second image, wherein the second image is a non-key frame, wherein the segmentation mask machine learning model is configured to generate a segmentation mask for a key frame; estimate a second set of forward motion vectors between the second image and another previous image; estimate the reliability of the second set of forward motion vectors; and determine to apply the segmentation mask machine learning model to the second image based on the estimation that the second set of forward motion vectors is unreliable.

[0126] Aspect 26: The non-transitory computer-readable medium of any one of claims 21 to 25, wherein the instructions further cause the at least one processor to mark one or more portions of the first segmentation mask as part of the foreground or background.

[0127] Aspect 27: A non-transitory computer-readable medium according to claim 26, wherein the instructions further cause the at least one processor to: detect an uncovered portion of the first image based on the first set of forward motion vectors; and fill the uncovered portion based on adjacent background portions of the first segmentation mask and the previous image.

[0128] Aspect 28: A non-transitory computer-readable medium according to claim 27, wherein, in order to fill the uncovered portion, the instructions further cause the at least one processor to: identify a motion vector associated with a foreground portion of the first segmentation mask that is closest to a portion of the uncovered portion; copy the motion vector; and associate the copied motion vector with the portion of the uncovered portion.

[0129] Aspect 29: A non-transitory computer-readable medium according to any one of claims 21 to 28, wherein the first image is a non-keyframe, and wherein the segmentation mask machine learning model is configured to generate a segmentation mask for a keyframe.

[0130] Aspect 30: A non-transitory computer-readable medium according to any one of claims 21 to 29, wherein the instructions further cause the at least one processor to: receive a third image, wherein the third image is a key frame, wherein the segmentation mask machine learning model is configured to generate a segmentation mask for the key frame; and apply the segmentation mask machine learning model to the third image.

[0131] Aspect 31: An apparatus comprising one or more components for performing the operations according to any one or more of Aspects 1 to 10.

Claims

1. A method for processing video data, the method comprising: receiving a first image subsequent to a previous image; receiving a first segmentation mask associated with the previous image; estimating a first set of forward motion vectors between the previous image and the first image; estimating the reliability of said first set of forward motion vectors; as well as Based on the estimated reliability of the first set of forward motion vectors, a second segmentation mask associated with the first image is extrapolated using the first set of forward motion vectors and the first segmentation mask.

2. The method of claim 1 , wherein estimating the reliability of the first set of forward motion vectors comprises: determining one or more heuristic statistics based on said first set of forward motion vectors; as well as The one or more heuristic statistics are evaluated to estimate the reliability of the first set of forward motion vectors.

3. The method of claim 2, wherein the one or more heuristic statistics include at least one of: the gradient of one or more motion vectors of said first set of forward motion vectors, or Percent change in outlook.

4. The method of claim 3, wherein evaluating the one or more heuristic statistics comprises at least one of: evaluating the gradient for one or more motion vectors against a gradient threshold, or The percentage change of the foreground is estimated based on a threshold of the foreground motion vector.

5. The method according to claim 1, further comprising: receiving a second image, wherein the second image is a non-keyframe, wherein the segmentation mask machine learning model is configured to generate a segmentation mask for the keyframe; estimating a second set of forward motion vectors between the second image and another previous image; estimating reliability of said second set of forward motion vectors; as well as Based on estimating that the second set of forward motion vectors is unreliable, determining to apply a segmentation mask machine learning model to the second image. 6 . The method of claim 1 , further comprising marking one or more portions of the first segmentation mask as being part of the foreground or background.

7. The method according to claim 6, further comprising: detecting an uncovered portion of the first image based on the first set of forward motion vectors; as well as The uncovered portion is filled based on an adjacent background portion of the first segmentation mask and the previous image.

8. The method of claim 7, wherein filling the uncovered portion comprises: identifying a motion vector associated with a foreground portion of the first segmentation mask that is closest to a portion of the uncovered portion; copying the motion vector; as well as The copied motion vector is associated with the portion of the uncovered portion.

9. The method of claim 1, wherein the first image is a non-keyframe, and wherein the segmentation mask machine learning model is configured to generate a segmentation mask for a keyframe.

10. The method according to claim 1, further comprising: receiving a third image, wherein the third image is a keyframe, wherein the segmentation mask machine learning model is configured to generate a segmentation mask for the keyframe; as well as Applying a segmentation mask machine learning model to the third image.

11. A device for processing video data, the device comprising: at least one memory; and at least one processor, the at least one processor coupled to the at least one memory, the at least one processor configured to: receiving a first image subsequent to a previous image; receiving a first segmentation mask associated with the previous image; estimating a first set of forward motion vectors between the previous image and the first image; estimating the reliability of said first set of forward motion vectors; as well as Based on the estimated reliability of the first set of forward motion vectors, a second segmentation mask associated with the first image is extrapolated using the first set of forward motion vectors and the first segmentation mask.

12. The apparatus of claim 11, wherein to estimate the reliability of the first set of forward motion vectors, the at least one processor is further configured to: determining one or more heuristic statistics based on the first set of forward motion vectors; and The one or more heuristic statistics are evaluated to estimate the reliability of the first set of forward motion vectors.

13. The apparatus of claim 12, wherein the one or more heuristic statistics include at least one of: the gradient of one or more motion vectors of said first set of forward motion vectors, or Percent change in outlook.

14. The apparatus of claim 13, wherein to evaluate the one or more heuristic statistics, the at least one processor is further configured to: evaluating the gradient for one or more motion vectors against a gradient threshold, or The percentage change of the foreground is estimated based on a threshold of the foreground motion vector.

15. The apparatus of claim 11, wherein the at least one processor is further configured to: receiving a second image, wherein the second image is a non-keyframe, wherein the segmentation mask machine learning model is configured to generate a segmentation mask for the keyframe; estimating a second set of forward motion vectors between the second image and another previous image; estimating reliability of said second set of forward motion vectors; as well as Based on estimating that the second set of forward motion vectors is unreliable, determining to apply a segmentation mask machine learning model to the second image.

16. The apparatus of claim 11, wherein the at least one processor is further configured to label one or more portions of the first segmentation mask as part of a foreground or a background.

17. The apparatus of claim 16, wherein the at least one processor is further configured to: detecting an uncovered portion of the first image based on the first set of forward motion vectors; and The uncovered portion is filled based on an adjacent background portion of the first segmentation mask and the previous image.

18. The apparatus of claim 17, wherein to fill the uncovered portion, the at least one processor is further configured to: identifying a motion vector associated with a foreground portion of the first segmentation mask that is closest to a portion of the uncovered portion; copying the motion vector; and The copied motion vector is associated with the portion of the uncovered portion.

19. The apparatus of claim 17, wherein the first image is a non-keyframe, and wherein the segmentation mask machine learning model is configured to generate a segmentation mask for a keyframe.

20. The apparatus of claim 11, wherein the at least one processor is further configured to: receiving a third image, wherein the third image is a keyframe, wherein the segmentation mask machine learning model is configured to generate a segmentation mask for the keyframe; and Applying a segmentation mask machine learning model to the third image.

21. A non-transitory computer readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to: receiving a first image subsequent to a previous image; receiving a first segmentation mask associated with the previous image; estimating a first set of forward motion vectors between the previous image and the first image; estimating the reliability of said first set of forward motion vectors; as well as Based on the estimated reliability of the first set of forward motion vectors, a second segmentation mask associated with the first image is extrapolated using the first set of forward motion vectors and the first segmentation mask.

22. The non-transitory computer readable medium of claim 21, wherein to estimate the reliability of the first set of forward motion vectors, the instructions further cause the at least one processor to: determining one or more heuristic statistics based on the first set of forward motion vectors; and The one or more heuristic statistics are evaluated to estimate the reliability of the first set of forward motion vectors.

23. The non-transitory computer readable medium of claim 22, wherein the one or more heuristic statistics include at least one of: the gradient of one or more motion vectors of said first set of forward motion vectors, or Percent change in outlook.

24. The non-transitory computer readable medium of claim 23, wherein to evaluate the one or more heuristic statistics, the instructions further cause the at least one processor to: evaluating the gradient for one or more motion vectors against a gradient threshold, or The percentage change of the foreground is estimated based on a threshold of the foreground motion vector.

25. The non-transitory computer readable medium of claim 21, wherein the instructions further cause the at least one processor to: receiving a second image, wherein the second image is a non-keyframe, wherein the segmentation mask machine learning model is configured to generate a segmentation mask for the keyframe; estimating a second set of forward motion vectors between the second image and another previous image; estimating reliability of said second set of forward motion vectors; as well as Based on estimating that the second set of forward motion vectors is unreliable, determining to apply a segmentation mask machine learning model to the second image.

26. The non-transitory computer readable medium of claim 21, wherein the instructions further cause the at least one processor to mark one or more portions of the first segmentation mask as part of a foreground or a background.

27. The non-transitory computer readable medium of claim 26, wherein the instructions further cause the at least one processor to: detecting an uncovered portion of the first image based on the first set of forward motion vectors; and The uncovered portion is filled based on an adjacent background portion of the first segmentation mask and the previous image.

28. The non-transitory computer readable medium of claim 27, wherein to fill the uncovered portion, the instructions further cause the at least one processor to: identifying a motion vector associated with a foreground portion of the first segmentation mask that is closest to a portion of the uncovered portion; copying the motion vector; and The copied motion vector is associated with the portion of the uncovered portion.

29. The non-transitory computer-readable medium of claim 27, wherein the first image is a non-keyframe, and wherein the segmentation mask machine learning model is configured to generate a segmentation mask for a keyframe.

30. The non-transitory computer readable medium of claim 21, wherein the instructions further cause the at least one processor to: receiving a third image, wherein the third image is a keyframe, wherein the segmentation mask machine learning model is configured to generate a segmentation mask for the keyframe; and Applying a segmentation mask machine learning model to the third image.

Images