REMOVING DISTORTION FROM REAL-TIME VIDEO USING A MASKED SINGLE FRAME

Abstract

This document describes systems and techniques for removing distortion from real-time video using a masked frame. In some aspects, an image capture device including a video processing manager is configured to capture a video segment comprising a sequence of frames. The sequence of frames includes at least one current frame having a foreground and a background. The video processing manager receives an object mask, motion vectors, and a predicted mask for the current frame. The video processing manager generates a final mask for the current frame based on the object mask, the motion vectors, and the predicted mask. The video processing manager applies the final mask to the current frame to segment the foreground from the background and provide a masked frame.The video processing manager processes the masked frame to remove distortions to produce an output frame and outputs the output frame. By repeating the described process for each frame in the sequence of frames, the video processing manager provides an enhanced video segment.

Description

CROSS-REFERENCE TO RELATED REVELATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 377,484, filed September 28, 2022, which is incorporated herein by reference. STATE OF THE ART

[0002] Many video applications apply changes to a video segment globally. That is, for a video segment that includes a foreground and a background, the changes are applied equally to both the foreground and the background. For example, motion blur can be applied globally to a video segment to hide judder artifacts resulting from a three-two pulldown or a fast-panning action shot. However, for video segments that include prominent foregrounds, applying motion blur globally results in blurring of the prominent foregrounds, which can be undesirable. SUMMARY

[0003] This document describes systems and techniques for removing distortion from real-time video using a masked frame. In some aspects, an image capture device having a video processing manager is configured to receive a video segment comprising a sequence of frames. The sequence of frames includes at least one current frame having a foreground and a background. The video processing manager receives an object mask, motion vectors, and a predicted mask for the current frame. The video processing manager generates a final mask for the current frame based on the object mask, the motion vectors, and the predicted mask. The video processing manager then applies the final mask to the current frame to segment the foreground from the background and provide a masked frame.The video processing manager processes the masked frame to remove distortions to produce an output frame and outputs the output frame. By repeating the described process for each frame in the sequence of frames, the video processing manager provides an enhanced video segment with less distortion.

[0004] Details of one or more aspects of removing distortion from real-time video using a masked frame are set forth in the accompanying drawings and the following description. Additional features and advantages are set forth in the drawings, description, and claims. This summary is provided to introduce subject matter further described in the detailed description and drawings. Accordingly, this summary neither describes essential features nor limits the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] This specification describes systems and techniques for removing distortion from real-time video using a masked frame with reference to the following drawings, in which the same number is used in all drawings to refer to similar features and components: Fig. 1 illustrates an example environment in which an image capture device may implement aspects of removing distortion from real-time video using a masked frame; Fig. 2 illustrates an example implementation of the image capture device of Fig. 1 in detail; Fig. 3 illustrates an example method for extracting a foreground from a video segment according to one or more aspects; and Fig. 4 illustrates an example method for segmenting a current frame of a sequence of frames to create a segmentation result. DETAILED DESCRIPTIONOverview

[0006] Ever since the advent of video cameras, engineers have strived to improve their capabilities so that the videos they capture appear lifelike. To this end, engineers have focused on improving resolution and frame rate capabilities. Unlike displays, reality is not limited to a finite resolution consisting of a finite number of individual pixels or light sources. Rather, reality can be thought of as having infinite resolution. A camera capable of capturing high-resolution video of a scene is therefore important for a lifelike representation of the scene. Also, unlike displays, reality is not limited to a finite frame rate consisting of a finite number of individual images or still pictures presented in rapid succession. Rather, reality can be thought of as having an infinite frame rate.A camera capable of capturing video of a scene at a very high frame rate is therefore essential for a lifelike representation of the scene. Given unlimited resources, space, and time, developing a camera with such capabilities is trivial. However, without unlimited resources, space, and time, engineers have developed alternative solutions.

[0007] For example, a parent attends a track and field event for their child, where the child is scheduled to compete in a 100-meter (100-yd) dash. They line up at the starting line and stretch their legs in preparation. Before the starting pistol fires, the parent opens a camera application on a smartphone and selects a video mode set to capture a 1080p recording at 30 frames per second (fps). When the starting pistol fires, the child launches off down the race track like a rocket. Meanwhile, the parent begins capturing a video segment using the smartphone. The parent quickly pans the camera to keep the child in focus and centered in the frame as they race past the parent toward the finish line. As they approach the finish line, they lunge forward with their shoulders and head, narrowly crossing the finish line.

[0008] Once the child has recovered from their 100-meter dash, the parent and child view the video segment of the child's performance. With many conventional approaches, the video contains errors, such as blur in the foreground, including the child, and a background that includes stalls and other athletes. This may be due to a real-time video processing pipeline on the parent's smartphone applying global motion blur to the foreground and background of the video segment to hide judder caused by the parent's rapid panning of the smartphone. Without the motion blur, the background of the video segment would contain judder or a stuttering artifact resulting from the rapidly panning action shot. Although the motion blur reduces the judder, the foreground, which includes the parent's child, is also blurred.This quality of the video is an undesirable result for both parents and children.

[0009] This document describes systems and techniques for removing distortion from real-time video using a masked frame. The disclosed systems and techniques can address a blurred foreground in a video segment resulting from a global application of motion blur. The systems and techniques extract the foreground from the video segment, thereby separating the foreground from the background and enabling editing of the background separately from the foreground. Operating environments, techniques that may be employed in the operating environments, and example methods are described below. Although systems and techniques are described that are directed to removing distortion from real-time video using a masked frame, the subject matter of the appended claims is not limited to the specific features or methods described.Rather, the specific features and methods are disclosed as an example implementation and reference is made to the operating environment only by way of example. Example environment

[0010] Fig. 1 illustrates an example environment 100 in which an image capture device 102 can implement aspects for removing distortion from real-time video using a masked frame. The image capture device 102 includes a display 104, a camera 106, one or more processors 108, and a video processing manager 110 configured to extract the foreground of a video segment in real time. In one example, a user 112 wishes to capture a video of an athlete 114 sprinting past a tree 116. The user 112 removes the image capture device 102, opens an application (not shown) installed on the image capture device 102, selects a video mode (not shown), and presses a shutter button (not shown) to start recording the athlete 114 sprinting past the tree 116.

[0011] In response to user 112 pressing the shutter button, video processing manager 110 captures a video segment comprising a sequence of frames including a previous frame and a current frame received immediately after the first frame. Video processing manager 110 receives an object mask for the current frame. The object mask may be generated, for example, using a machine-learned (ML) model on image capture device 102. Video processing manager 110 receives motion vectors for the current frame, which may be generated by an optical flow measurement tool on image capture device 102. The optical flow measurement tool may, for example, generate the motion vectors using the previous and current frames. Video processing manager 110 receives a predicted mask for the current frame.The predicted mask may be generated from the motion vectors and the previous frame. For example, the video processing manager 110 may modify (e.g., move, scale, rotate) a mask for the previous frame according to the motion vectors for the current frame. The video processing manager 110 generates a final mask for the current frame based on the object mask, the motion vectors, and the predicted mask. The video processing manager 110 then applies the final mask to the current frame to provide a masked frame for which a foreground and a background are segmented from each other. The video processing manager 110 processes the masked frame to remove distortion from the masked frame and generate an output frame.For example, the distortion may be a judder in the background of the masked frame, and the edits applied by video processing manager 110 may be motion blur. Video processing manager 110 may apply motion blur to the background of the masked frame to hide the judder. Next, video processing manager 110 outputs the output frame.

[0012] The video processing manager 110 may repeat the method described herein for a second current frame with respect to the current frame as the second previous frame. For example, a sequence of frames includes a first frame, a second frame, a third frame, and a fourth frame. For a first iteration of the method, the first frame is the previous frame and the second frame is the current frame. The video processing manager may implement one or more portions of the method to generate a final mask for the second frame. For a second iteration of the method, the second frame is the second previous frame and the third frame is the second current frame. In the context of the second iteration, the second frame is the previous frame and the third frame is the current frame.For a third iteration of the method, the third frame is the second previous frame and the fourth frame is the second current frame. That is, within the context of the third iteration, the third frame is the previous frame and the fourth frame is the current frame. Although four frames were described in the present example, the sequence of frames can contain any number of frames, and the video processing manager can iterate through previous, current, second previous, and second current frames until a final mask is generated for each frame of the sequence of any number of frames.

[0013] After the video segment of the athlete 114 in the foreground sprinting past the tree 116 in the background has been recorded, the user 112 reviews the video segment. As in Fig. 1, a display 104-1 illustrates a first frame of a sequence of three or more frames. The sprinting athlete 114 is centered in the foreground of the first frame. The tree 116-1 is located in the background on the right side of the first frame. As illustrated, the tree 116-1 includes a dark front surface and two lighter surfaces on the left side. The two lighter surfaces of the tree 116-1 represent a motion blur applied to the background of the first frame.

[0014] Further, a second frame of the sequence of three or more frames is illustrated by a display 104-2. Again, athlete 114, centered in the foreground of the second frame, sprints past tree 116-2, centered in the background of the second frame. Tree 116-2 includes the dark front surface and two lighter surfaces to its left, representing a motion blur applied to the background of the second frame.

[0015] The display of 104-3 further illustrates a third frame of the sequence of three or more frames. Athlete 114 remains centered in the foreground of the third frame. Athlete 114 continues to sprint past tree 116-3 to the left of a background of the third frame. Again, tree 116-3 includes the dark foreground surface and two lighter surfaces to the left to represent motion blur applied to the background of the third frame. User 112 is satisfied with the video segment because video processing manager 110 extracted the foreground from the video segment in real time and applied motion blur only to the background to hide the judder in the background. Although judder is described, it can be any type of distortion.Although processing the background of a video segment is described, the video processing manager 110 may similarly implement any of the disclosed systems and techniques to process the foreground of a video segment. Example implementations

[0016] Fig. 2 illustrates an example implementation 200 of the image capture device 102 of Fig. 1 in detail. The image capture device 102 is illustrated as a variety of example devices, including consumer electronics devices. As non-limiting examples, the image capture device 102 may be a smartphone 102-1, a tablet 102-2, a laptop 102-3, a desktop computer 102-4, a smartwatch 102-5, smart glasses 102-6, a game controller 102-7, a speaker 102-8, or a microwave oven 102-9. Although not shown, the image capture device 102 may also be implemented as an audio recording device, condition monitoring device, home automation system, home security system, game console, personal media device, personal assistant device, drone, household appliance, and the like. It should be noted that the image capture device 102 may be portable, non-portable but mobile, or relatively immobile (e.g., desktop computers, household appliances).It should also be noted that the image capture device 102 may be used with or embedded in many image capture devices 102 or peripheral devices, such as those found in motor vehicles or connected to a personal computer. The image capture device 102 may include additional components shown in FIG. 1 for clarity. Fig. 2 omitted components and interfaces.

[0017] As in Fig. 2, the image capture device 102 includes a display 104, a camera 106, and processors 108. The display 104 may be any of a variety of displays, including a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, an in-plane switching (IPS) display, a twisted nematic (TN) liquid crystal display, and so on. The display may be referred to as a "screen" so that content (e.g., images, videos) may be displayed "on screen." The camera 106 may include one or more image sensors, one or more lenses, one or more autofocus motors, a flash, image stabilization components, and so on. The camera 106 can be configured to record videos at different resolutions (e.g., 1080p, 2k, 4k) and frame rates (e.g.,30 fps, 60 fps, 120 fps). The camera 106 may include an associated application with which a user can interact to adjust capture settings (e.g., resolution, frame rate) and review captured images and videos. The processors 108 may include one or more suitable single-core or multi-core processors, such as a graphics processing unit (GPU) or a central processing unit (CPU).

[0018] The image capture device 102 includes a computer-readable medium (CRM) 202, which is also Fig. 2. The CRM 202 includes a storage medium 204 and a storage medium 206. The storage media 204 and storage media 206 may include one or more non-volatile storage devices, such as random-access memory (RAM), dynamic random-access memory (DRAM), a solid-state drive (SSD), a magnetic rotating hard disk drive (HDD), or any other type of storage media suitable for storing electronic instructions, each coupled to a data bus. The term "coupled" may refer to two or more elements that are in direct contact (e.g., physical, electrical, magnetic, optical) or to two or more elements that are not in direct contact with each other but nevertheless cooperate or interact with each other.

[0019] The CRM 202 further includes an operating system (OS) 208, applications 210, and a video processing manager 110. The OS 208, applications 210, and video processing manager 110 may be implemented as computer-readable instructions on the CRM 202 that may be executed by the processors 108 to provide some or all of the functionality described herein. For example, the processors 108 may perform certain computational tasks of the OS 208 aimed at removing distortion from real-time video using a masked frame. The applications 210 may include power management applications, camera applications, background applications, applications for communications (e.g., audio and video calls), and so on.

[0020] In some aspects, implementations of the video processing manager 110 may include one or more integrated circuits (ICs), a system-on-a-chip (SoC), a secure key store, embedded hardware with firmware stored on a read-only memory (ROM), a printed circuit board (PCB) with various hardware components, or any combination thereof. As described herein, a system for removing distortion from real-time video using a masked frame may include one or more components of the image capture device 102, as shown in Fig. 1 and Fig. 2, which are configured to remove distortion from real-time videos using a masked frame. Additionally, the system for removing distortion from real-time videos using a masked frame may be implemented as image capture device 102.

[0021] Furthermore, Fig. 2, the image capture device 102 includes input / output (I / O) ports 212. The I / O ports 212 enable the image capture device 102 to interact with other devices or users via peripherals transmitting any combination of digital, analog, or radio frequency signals. The I / O ports 212 may include any combination of internal or external ports, such as Universal Serial Bus (USB) ports, audio ports, video ports, dual inline memory module (DIMM) card slots, peripheral component interconnect express (PCIe) slots, and so on. Various peripheral devices may be operatively coupled to the I / O ports 212, such as human input devices (HIDs), external CRMs, speakers, displays, keyboards, mice, or other peripherals.Although not shown, the image capture device 102 may also include a system bus, interconnect, or data transmission system coupled to the various components within the image capture device 102. A system bus or interconnect may include one or a combination of different bus structures, such as a memory bus, a peripheral bus, a USB bus, a local bus, or a processor bus utilizing one of several bus architectures.

[0022] Furthermore, the image capture device 102 includes one or more sensors 214, as shown in Fig. 2. The sensors 214 may be disposed anywhere on or within the image capture device 102. Additionally or alternately, the sensors 214 may be disposed on or within a peripheral device connected to the image capture device 102 (e.g., wirelessly, wired). The sensors 214 may include a variety of components, such as an audio sensor (e.g., a microphone), a touch sensor (e.g., a touchscreen), an image sensor (e.g., a phase-detection autofocus sensor that is part of a camera or camera system), an ambient light sensor (e.g., a photodetector), an acceleration sensor (e.g., an accelerometer), a proximity sensor (e.g., a laser autofocus sensor), or a pressure sensor (e.g., a barometer). The sensing components may be disposed within a housing of the image capture device 102.In implementations, the image capture device may include more than one or more of the sensing components. Example procedure

[0023] The following section describes example methods that the video processing manager 110 may perform in Fig. 1 and Fig. 2 to implement aspects of removing distortion from real-time video using a masked frame. The methods are shown as sets of blocks that specify operations or acts performed by the video processing manager 110, processors 108, sensors 214, or other unmentioned components of the image capture device. The methods are not limited to the order or combinations of the block sets shown for performing the operations by the respective blocks. Further, one or more of the operations may be repeated, combined, reorganized, or linked to provide additional or alternative methods. For example, in the following discussion, only the Fig. 1 and Fig. 2 detailed example implementations and entities are referred to.

[0024] Fig. 3 illustrates an example method 300 for extracting a foreground from a video segment according to one or more aspects. At 302, a video processing manager (e.g., video processing manager 110) receives a video segment. The video processing manager may capture the video segment using an image capture device (e.g., image capture device 102), components thereof (e.g., camera 106, sensors 214), or a combination thereof. For example, the video processing manager may utilize a camera of an image capture device to capture the video segment. The video segment consists of a sequence of frames that includes a previous frame and a current frame received immediately after the previous frame. For example, a video segment may include ten frames, the first of which is the previous frame.Accordingly, the second frame immediately following the first frame is the current frame.

[0025] At 304, the video processing manager receives an object mask for the current frame. The object mask may be generated using a machine learning (ML) model. The ML model may be trained using tagged objects, such as people, vehicles, pets, or other objects of interest that may be in the foreground of a video segment. Additionally, the ML model may be stored on an image capture device, for example, as computer-readable instructions stored on a CRM (e.g., CRM 202) of the image capture device. At 306, the video processing manager receives motion vectors for the current frame. The motion vectors may be generated by an optical flow measurement tool (e.g., an inverse compositional implementation of the Lucas-Kanade method) using the previous and current frames.The optical flow measurement tool may, for example, be stored as computer-readable instructions on a CRM of an image capture device. The motion vectors generated by the optical flow measurement tool describe a change in the position of a pixel or group of pixels from the previous frame to the current frame. The motion vectors may, for example, be stored as a heatmap or other suitable encoding on a CRM (e.g., CRM 202) of an image capture device. At 308, the video processing manager receives a predicted mask for the current frame. The predicted mask is generated from the motion vectors and the previous frame. For example, a mask for the previous frame may be aligned (e.g., rotated, translated, scaled) to the current frame using the motion vectors.

[0026] At 310, the video processing manager generates a final mask for the current frame. The final mask is based on the object mask, the motion vectors, and the predicted mask. For example, the video processing manager may generate the final mask by combining the object mask and the predicted mask. Although not shown, the video processing manager may apply a sharpening operation to the final mask before proceeding to 312. The sharpening operation may be based on a luma version (e.g., grayscale) of the current frame, a bilateral mesh, and the final mask. The sharpening operation may sharpen the edge of the final mask to avoid a dull or rough final mask.

[0027] At 312, the video processing manager applies the end mask to the current frame to provide a masked frame. The end mask segments a foreground of the masked frame from a background of the masked frame. At 314, the video processing manager processes the masked frame to remove distortion from the masked frame and produce an output frame. For example, the distortion in the background of the masked frame may be judder. Judder may result from a three-two pulldown, low frame rate video capture (e.g., 24 fps, 30 fps), fast-panning video capture, or a combination thereof. By segmenting the foreground from the background of the masked frame using the end mask, the video processing manager enables the foreground and background of the masked frame to be processed separately.Accordingly, the video processing manager may apply motion blur exclusively to the background of the masked frame to hide judder. At 316, the video processing manager outputs the output frame. In this example, the output frame includes applying motion blur to the background and applying no edits to the foreground, thereby masking judder in the background and maintaining a sharp foreground. In implementations, the video processing manager may apply edits to the foreground, the background, both the foreground and the background, or neither the foreground nor the background. Further, the edits may include one or more of a variety of edits, including, but not limited to, cropping, color adjustments, highlight adjustments, filter applications, Gaussian blur, or motion blur.

[0028] Fig. 4 illustrates an example method 400 for segmenting a current frame of a sequence of frames to create a segmentation result. Each of the blocks illustrated in example method 400 may be repeated, combined, reorganized, or connected to sets of blocks in example method 400 or example method 300. For example, example method 400 may utilize the motion vectors received at 306 of example method 300.

[0029] At 402, a video processing manager quantizes motion vectors for the current frame into two or more bins. The two or more bins may contain one or more motion vectors per bin. The bins group similar motion vectors together, which may be used in step 404.

[0030] At 404, the video processing manager calculates an average motion vector for a bin of the two or more bins that contains a majority of the motion vectors. As an example, reference is made to the example environment 100 of Fig. 1. Since the user 112 has panned the image capture device 102 to keep the athlete 114 centered in the foreground of the video segment, the motion vectors may be grouped into two bins. A first bin contains the motion vectors for the background, and a second bin contains the motion vectors for the foreground. Further, the motion vector bin in the background may be the bin containing the majority of motion vectors because the athlete 114 occupies less space in each frame of the video segment. Accordingly, the video processing manager may calculate the average motion vector for the background bin.

[0031] At 406, the video processing manager compares the motion vectors of the two or more bins to the average motion vector to create a comparison result. In the present example, the average motion vector for the bin in the background is greater than any of the motion vectors in the foreground. This disparity is due to the user 112 panning the image capture device 102 to keep the athlete 114 in the foreground centered. Relative to the image capture device 102, the athlete 114 is not moving. In other words, the motion vectors in the foreground are close to zero. In contrast to the athlete 114 in the foreground, the tree 116 in the background is moving relative to the image capture device 102. In other words, the motion vectors in the background are greater than zero.In the present example, the comparison result may indicate that the motion vectors in the foreground are smaller than the average motion vector of the background motion vectors.

[0032] At 408, the video processing manager classifies one or more of the motion vectors as outliers based on the comparison result exceeding a threshold. The threshold may be an integer, a fraction, a percentage, a difference relative to another value (e.g., the average motion vector), or another quantifier to which a motion vector may be compared. In the present example, the video processing manager may classify the foreground motion vectors as outliers based on the fact that they are smaller than the average motion vector by a difference (e.g., 10 percent, 15 percent). As additional examples, the video processing manager may classify a motion vector as an outlier if it is larger than the average motion vector by a difference (e.g., 10 percent, 25 percent).As another example, the video processing manager may classify a motion vector as an outlier if it is close to zero, close to infinity, close to another integer, or close to another discrete value.

[0033] At 410, the video processing manager segments the current frame based on the outliers to create a segmentation result of the current frame. Continuing the present example, the segmentation result may include two segments, one for the foreground and one for the background. Based on the foreground segment and the segment in the background, the video processing manager may process the background separately from the foreground, the foreground separately from the background, or a combination of both. Further, in combination with example method 300, the video processing manager may generate the final mask by combining the segmentation result with the predicted mask.

[0034] In some aspects, the video processing manager may utilize distance information from additional sensors (e.g., sensors 214) of the image capture device. For example, the video processing manager may utilize distance information from a proximity sensor (e.g., sonar, radar, lidar) to more quickly or accurately identify a foreground or background of a previous or current frame. The distance information may include a distance measurement (e.g., 6 m, 15 m) for the foreground and a distance measurement (e.g., 25 m, 31 m) for the background. The video processing manager may segment the foreground of the current frame from the background of the current frame based on the distance information from the proximity sensor.As another example, the video processing manager may use a second camera that has a different viewpoint than the first camera to identify a foreground or background of a current frame. The background of the current frame may appear similar from the viewpoint of the first camera and the viewpoint of the second camera. The foreground of the current frame may appear different from the viewpoint of the first camera and the viewpoint of the second camera. The video processing manager may segment the foreground from the background of the current frame based on the differences in the foreground or similarities in the background from the different viewpoints.

[0035] Throughout this discussion, examples are provided where a video processing manager processes the background of a frame of a sequence of frames of a video segment. However, the systems and techniques described herein are not limited to processing the background of a frame. In certain aspects, the systems and techniques may also be implemented by a video processing manager to process the foreground of a frame. Additionally or alternatively, the systems and techniques described herein may be implemented by a video processing manager in a long-exposure photography application. For example, a user may want to take a photo of an object in low-light conditions.To do so, the user captures a shot of the object using an image capture device having a video processing manager configured to remove distortion from real-time video using a masked frame. The video processing manager may capture multiple low-light, long-exposure images of the object. The long-exposure provides sufficient time for an image sensor of the image capture device to capture enough light for each frame. If the user has shaky hands when capturing the multiple long-exposure frames, the object may be blurry. However, the video processing manager may implement the techniques and systems described herein to segment a background from a foreground of the multiple frames.For example, the video processing manager can use motion vectors to perform segmentation. The video processing manager can also use the motion vectors to stabilize the foreground of the long-exposure frames in real time (e.g., by aligning a foreground mask with the motion vectors to a current frame), resulting in a clear foreground. The multiple frames can be combined (e.g., overlaid), for example, into a single output photo that has a clear foreground. Additional examples

[0036] Additional examples are provided in the following section.

[0037] Example 1: A method comprising: receiving a video segment, wherein the video segment comprises a sequence of frames, wherein the sequence of frames includes a previous frame and a current frame, wherein the current frame immediately follows the previous frame; receiving an object mask for the current frame, wherein the object mask is generated using a machine learning (ML) model; receiving motion vectors for the current frame, wherein the motion vectors are generated by an optical flow measurement tool using the previous frame and the current frame; receiving a predicted mask for the current frame, wherein the predicted mask is generated from the motion vectors and the previous frame;Generating a final mask for the current frame, the final mask being generated based on the object mask, the motion vectors, and the predicted mask; applying the final mask to the current frame to provide a masked frame; processing the masked frame to remove distortion from the masked frame and generate an output frame; and outputting the output frame.;

[0038] Example 2: The method of example 1, wherein: the video segment is captured by and received from a camera of an image capture device; the ML model is located on the image capture device; and the optical flow measurement tool is located on the image capture device.

[0039] Example 3: The method of Example 1, further comprising: quantizing the motion vectors for the current frame into two or more bins; calculating an average motion vector for a bin of the two or more bins that contains a majority of the motion vectors; comparing the motion vectors of the two or more bins to the average motion vector to generate a comparison result; classifying one or more of the motion vectors as outliers based on the comparison result exceeding a threshold; and segmenting the current frame based on the outliers to generate a segmentation result of the current frame.

[0040] Example 4: The method of Example 3, wherein the final mask is generated by combining the segmentation result of the current frame with the predicted mask.

[0041] Example 5: The method of Example 1, wherein the predicted mask is generated by aligning an end mask of the previous frame to the current frame using the motion vectors.

[0042] Example 6: The method of Example 1, further comprising: performing a sharpening process on the end mask before applying the end mask to the current frame.

[0043] Example 7: The method of example 6, wherein: the sharpening process is performed by an edge sharpening tool; and the edge sharpening tool is located on an image capture device from which the video segment is received.

[0044] Example 8: The method of Example 1, wherein the previous and current frames include a background, a foreground in front of the background, and an object of interest in the foreground.

[0045] Example 9: The method of example 8, wherein processing the masked frame to generate the output image further comprises: processing the background of the masked frame.

[0046] Example 10: The method of example 8, wherein processing the masked frame to generate the output image further comprises: processing the foreground of the masked frame.

[0047] Example 11: The method of Example 8, wherein processing the masked frame to generate the output image further comprises: processing the foreground and the background of the masked frame.

[0048] Example 12: The method of Example 8, further comprising: receiving range information for the current image, wherein the range information is acquired by a sensor of an image acquisition device; and segmenting, based on the range information, the foreground of the current image frame from the background of the current image frame.

[0049] Example 13: The method of Example 8, further comprising: receiving viewpoint information for the current image, wherein the viewpoint information is acquired by a sensor of a different image acquisition device; and segmenting, based on the viewpoint information, the foreground of the current image frame from the background of the current image frame.

[0050] Example 14: An image capture device comprising: at least one camera; one or more sensors; one or more processors; and memory instructions that, when executed by the one or more processors, cause the one or more processors to implement a video processing manager to provide video processing utilizing the at least one camera, the one or more sensors, and the one or more processors by performing the method of any preceding claim.

[0051] Example 15: A computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 13. conclusion

[0052] Unless the context dictates otherwise, the use of the word "or" in this document may be considered the use of an "inclusive or" or an expression that allows the inclusion or application of one or more elements linked to the word "or" (e.g., a phrase "A or B" may be interpreted to allow only "A," to allow only "B," or to allow both "A" and "B"). In addition, as used in this document, a phrase that refers to "at least one of" a list of elements refers to any combination of those elements, including individual elements. For example, "at least one of a, b, or c" can cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other order of a, b, and c).Furthermore, elements shown in the accompanying drawings and terms discussed in this specification may identify one or more elements or terms, so that singular or plural forms of the elements and terms may be referred to interchangeably in this written description.

[0053] Although implementations of systems and techniques, as well as devices, that enable distortion removal from real-time video using a masked frame have been described in language specific to certain features and / or methods, the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as example implementations for distortion removal from real-time video using a masked frame. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 377,484

[0001]

Claims

[1] Method comprising: Receiving a video segment, the video segment comprising a sequence of frames, the sequence of frames including a previous frame and a current frame, the current frame immediately following the previous frame in the sequence; Receiving an object mask for the current frame, wherein the object mask is generated using a machine-learned (ML) model; Receiving motion vectors for the current frame, wherein the motion vectors are generated by an optical flow measurement tool using the previous frame and the current frame; Receiving a predicted mask for the current frame, the predicted mask being generated from the motion vectors and the previous frame; Generating a final mask for the current frame, the final mask being based on the object mask, the motion vectors, and the predicted mask; Apply the end mask to the current frame to provide a masked frame; Processing the masked frame to remove distortions from the masked frame and produce an output frame; and Output the output frame. [2] The method of claim 1, wherein: the video segment is captured and received by a camera of an image capture device; the ML model is stored on a computer-readable medium (CRM) of the image capture device; and the optical flow measurement tool is stored on the CRM of the image acquisition device. [3] The method of claim 1, further comprising: Quantizing the motion vectors for the current frame in two or more storage locations; Calculating an average motion vector for a storage location of the two or more storage locations that contains a majority of the motion vectors; Comparing the movement vectors of the two or more storage locations with the average movement vector to create a comparison result; Classifying one or more of the motion vectors as outliers based on the comparison result exceeding a threshold; and Segment the current frame based on the outliers to create a segmentation result of the current frame. [4] The method of claim 3, wherein the final mask is generated by combining the segmentation result of the current frame with the predicted mask. [5] The method of claim 1, wherein the predicted mask is generated by aligning an end mask of the previous frame to the current frame using the motion vectors. [6] The method of claim 1, further comprising: Perform a sharpening process on the end mask before applying the end mask to the current frame. [7] The method of claim 6, wherein: the sharpening process is carried out with an edge sharpening tool; and the edge sharpening tool is located on an image capture device from which the video segment is received. [8] The method of claim 1, wherein the previous and current frames include a background, a foreground in front of the background, and an object of interest in the foreground. [9] The method of claim 8, wherein processing the masked frame to generate the output frame further comprises: Edit the background of the masked frame. [10] The method of claim 8, wherein processing the masked frame to generate the output frame further comprises: Editing the foreground of the masked frame. [11] The method of claim 8, wherein processing the masked frame to generate the output frame further comprises: Edit the foreground and background of the masked frame. [12] The method of claim 8, further comprising: Receiving distance information for the current frame, wherein the distance information is acquired by a sensor of an image capture device; and Segment the foreground of the current frame from the background of the current frame based on the distance information. [13] The method of claim 8, further comprising: Receiving viewpoint information for the current frame, wherein the viewpoint information is acquired by a different image acquisition device; and Segment the foreground of the current frame from the background of the current frame based on the viewpoint information. [14] Image capture device comprising: at least one camera; one or more sensors; one or more processors; and Memory instructions that, when executed by the one or more processors, cause the one or more processors to implement a video processing manager to provide video processing utilizing the at least one camera, the one or more sensors, and the one or more processors by performing the method of any preceding claim. [15] A computer-readable medium (CRM) comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 13.

Images