Segmentation using monocular depth estimation

By employing monocular depth estimation and filtering techniques like adaptive Gaussian thresholding and intersection-over-union matching, the method improves image segmentation accuracy and stability, addressing the inconsistency issues in existing methods.

JP2024542653AInactive Publication Date: 2024-11-15QUALCOMM INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024532291
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2021-12-01
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing image processing techniques, particularly those using monocular depth estimation, often yield poor and inconsistent results in image segmentation, especially when dealing with objects or people that are distant or smaller in the captured scene, leading to inaccuracies and flickering in videos.

Method used

The use of monocular depth estimation to generate a depth map and filter background objects based on depth values, applying adaptive Gaussian thresholding and noise reduction, and performing intersection-over-union matching to improve segmentation accuracy and stability.

Benefits of technology

This approach enhances segmentation accuracy by filtering out background objects that are not of interest, reducing flickering and improving the consistency of segmentation results, particularly for distant or smaller objects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024542653000001_ABST
    Figure 2024542653000001_ABST
Patent Text Reader

Abstract

A system, method, and computer readable medium for performing image segmentation with depth filtering are provided. In some embodiments, the method can include obtaining a frame capturing a scene, generating a first segmentation map based on the frame, the first segmentation map including a target segmentation mask that identifies a target of interest and one or more background masks that identify one or more background regions of the frame, and generating a second segmentation map including the first segmentation map with the one or more background masks filtered, where the one or more background masks are filtered from the first segmentation map based on a depth map associated with the frame.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates generally to image processing. For example, aspects of the present disclosure relate to segmentation using monocular depth estimation. [Background technology]

[0002] The increasing versatility of digital camera products has allowed digital cameras to be incorporated into a wide variety of devices, expanding their use to a variety of applications. For example, phones, drones, automobiles, computers, televisions, and many other devices today are often equipped with camera devices. The camera devices allow users to capture images and / or videos from any system equipped with a camera device. The images and / or videos can be captured for entertainment use, professional photography, surveillance, and automation, among other applications. Furthermore, camera devices are increasingly equipped with specific features for modifying images or creating artistic effects on images. For example, many camera devices are equipped with image processing capabilities for creating different effects on the captured images.

[0003] Many implemented image processing techniques rely on image segmentation algorithms that divide an image into segments, which can then be analyzed or processed to identify objects, generate specific image effects, etc. Some exemplary practical applications of image segmentation include, but are not limited to, chromakey compositing, feature extraction, object detection, recognition tasks (e.g., object recognition, face recognition, etc.), image stylization, machine vision, medical imaging, and depth of field (or "bokeh") effects, among others. However, camera devices and image segmentation techniques often produce poor and inconsistent results. Summary of the Invention

[0004] Systems and techniques for improving the stability of segmentation using monocular depth estimation are described herein. According to at least one embodiment, a method of segmentation using monocular depth estimation is provided. An exemplary method can include obtaining a frame capturing a scene, generating a first segmentation map based on the frame, the first segmentation map including a target segmentation mask identifying a target of interest and one or more background masks identifying one or more background regions of the frame, and generating a second segmentation map including the first segmentation map with the one or more background masks filtered, where the one or more background masks are filtered from the first segmentation map based on a depth map associated with the frame.

[0005] According to at least one embodiment, a non-transitory computer readable medium for segmentation using monocular depth estimation is provided. An exemplary non-transitory computer readable medium can include instructions that, when executed by one or more processors, cause the one or more processors to obtain frames capturing a scene, generate a first segmentation map based on the frames, the first segmentation map including a target segmentation mask identifying a target of interest and one or more background masks identifying one or more background regions of the frames, and generate a second segmentation map including the first segmentation map with the one or more background masks filtered, where the one or more background masks are filtered from the first segmentation map based on a depth map associated with the frames.

[0006] According to at least one embodiment, an apparatus for segmentation using monocular depth estimation is provided. An exemplary apparatus may include a memory and one or more processors coupled to the memory, where the one or more processors are configured to: obtain frames capturing a scene, generate a first segmentation map based on the frames, the first segmentation map including a target segmentation mask identifying a target of interest and one or more background masks identifying one or more background regions of the frames, and generate a second segmentation map including the first segmentation map with the one or more background masks filtered, where the one or more background masks are filtered from the first segmentation map based on a depth map associated with the frames.

[0007] According to at least one embodiment, another apparatus for segmentation using monocular depth estimation is provided, the apparatus can include means for obtaining frames capturing a scene, generating a first segmentation map based on the frames, the first segmentation map including a target segmentation mask identifying a target of interest and one or more background masks identifying one or more background regions of the frames, and generating a second segmentation map including the first segmentation map with the one or more background masks filtered, where the one or more background masks are filtered from the first segmentation map based on a depth map associated with the frames.

[0008] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include generating the depth map using a neural network.

[0009] In some examples, generating the second segmentation map may include determining a threshold difference between respective depth values ​​associated with the one or more background masks and respective depth values ​​associated with a target segmentation mask that identifies the target of interest based on a comparison of the first segmentation map and the depth map. In some examples, the second segmentation map further includes removing the one or more background masks from the first segmentation map based on a threshold difference between respective depth values ​​associated with the one or more background masks and respective depth values ​​associated with the target segmentation mask that identifies the target of interest.

[0010] In some examples, the depth map can include a set of depth masks associated with depth values ​​corresponding to pixels of the frame. In some aspects, generating the second segmentation map can include determining an overlap between a target segmentation mask identifying a target of interest and one or more depth masks from the set of depth masks in the depth map based on a comparison of the first segmentation map and the depth map, retaining the target segmentation mask identifying the target of interest based on the overlap, and filtering the one or more background masks from the first segmentation map based on additional overlap between the one or more background masks and the one or more additional depth masks from the set of depth masks.

[0011] In some aspects, generating the second segmentation map further includes determining that a difference between a depth value associated with the one or more additional depth masks and a depth value associated with the one or more depth masks exceeds a threshold, and filtering one or more background masks from the first segmentation map based on the difference exceeding the threshold. In some embodiments, the one or more depth masks correspond to a target of interest and the one or more additional depth masks correspond to one or more background regions of the frame.

[0012] In some aspects, generating the second segmentation map can include determining an intersection-over-union (IOU) score associated with a depth region from the depth map and a predicted mask from the first segmentation map; matching the depth region from the depth map with a predicted mask from the first segmentation map based on the IOU score, which may include a target segmentation mask identifying a target of interest and one or more background masks identifying one or more background regions of the frame; and filtering the one or more background masks from the first segmentation map based on a determination that the one or more IOU scores associated with the one or more background masks are below a threshold.

[0013] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include applying adaptive Gaussian thresholding and noise reduction to the depth map prior to filtering one or more background masks from the first segmentation map.

[0014] In some examples, the frames may include monocular frames generated by a monocular image capture device.

[0015] In some embodiments, the first segmentation map and the second segmentation map are generated using one or more neural networks.

[0016] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include generating a modified frame based on the frame and the second segmentation map. In some examples, the modified frame may include at least one of a visual effect, an extended reality effect, an image processing effect, a blur effect, an image recognition effect, an object detection effect, a computer graphics effect, a chromakeying effect, and an image stylization effect.

[0017] In some aspects, each of the above-mentioned devices may be, be part of, or include a mobile, device, smart or connected device, camera system, and / or extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device). In some examples, the device may include or be part of a vehicle, a mobile device (e.g., a mobile phone or a so-called "smartphone" or other mobile device), a wearable device, a personal computer, a laptop computer, a tablet computer, a server computer, a robotics device or system, an aviation system, or other device. In some aspects, the device includes an image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, the device includes one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device includes one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, the devices described above may include one or more sensors, which in some cases may be used to determine the location of the device, the state of the device (e.g., tracking state, operating state, temperature, humidity level, and / or other state), and / or for other purposes.

[0018] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0019] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0020] Exemplary embodiments of the present application are described in detail below with reference to the following drawings: [Brief description of the drawings]

[0021] [Figure 1] 1 is a block diagram illustrating an example image processing system, in accordance with some embodiments of the present disclosure. [Diagram 2] 1 illustrates an example scene with multiple objects in the background, in accordance with some embodiments of the present disclosure. [Diagram 3] FIG. 2 illustrates an example process for segmentation using depth estimation according to some embodiments of the present disclosure. [Figure 4] FIG. 2 illustrates an example depth filtering process for generating a segmentation output based on a segmentation map and estimated depth information, in accordance with some embodiments of the present disclosure. [Diagram 5] FIG. 1 illustrates example training and inference stages for segmentation using depth filtering, according to some embodiments of the present disclosure. [Figure 6] 3A-3C are diagrams illustrating examples of segmented frames without and with depth filtering, in accordance with some embodiments of the present disclosure. [Figure 7] 1 is a flowchart of an embodiment of a process for semantic segmentation using depth filtering, in accordance with some embodiments of the present disclosure. [Figure 8] 1 illustrates an exemplary computing device architecture in accordance with some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Specific aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting.

[0023] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0024] As mentioned above, computing devices are increasingly equipped with the ability to capture images, perform various image processing tasks, generate various image effects, and the like. Many image processing tasks and effects, such as chromakeying, depth of field or "bokeh" effects, object detection, recognition tasks (e.g., object, face, and biometric recognition), feature extraction, background replacement, image stylization, automation, machine vision, computer graphics, medical imaging, and the like, rely on image segmentation to divide an image into segments that can be analyzed or processed to perform a desired image processing task or generate a desired image effect. For example, cameras often include portrait mode capabilities that allow for a shallow depth of field ("bokeh") effect. A depth of field effect can focus on a particular image region or object, such as a foreground object or region, while blurring other regions or pixels in the image, such as background regions or pixels. A depth of field effect can be generated using image segmentation techniques to identify and modify different regions or objects in an image, such as background and foreground regions or objects.

[0025] In some cases, a user may only be interested in specific foreground objects in an image and / or video. For example, a user may only be interested in foreground objects that are close to the user, such as when the user takes a selfie of himself or a small group of people. As another example, a user may be interested in a specific foreground target in a video stream or video recording, personal media production, presentation, etc. In some cases, the device may be configured to perform object-based processing, where objects of interest are identified using semantic segmentation and optionally augmented by post-processing.

[0026] Semantic segmentation can generate a class mapping for each pixel of the view, and objects in the class, such as people, are identified from the image data. In many cases, the accuracy of semantic segmentation can be reduced when there are more distant people or objects in the captured scene (e.g., relative to closer people or objects of interest, such as foreground targets). The reduced accuracy can be caused by smaller size and / or smaller resolution of people or objects that are more distant in the scene. Inaccuracies and / or inconsistencies in semantic segmentation can cause artifacts and / or flickering in the video when remote people and / or objects (e.g., more distant people and / or objects in the captured scene and / or background) are included in the captured scene. Accurate semantic segmentation can detect and segment people and / or objects in the background that are not of interest, as well as the target(s) of interest, such as foreground objects.

[0027] In the following disclosure, systems, apparatus, methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and techniques") for improving segmentation stability using monocular depth estimation are described herein. In some examples, the systems and techniques described herein can generate a segmentation output, such as a segmentation map, that includes targets of interest and excludes any objects and / or people in the background that are not of interest. The systems and techniques described herein can generate segmentation maps and depth maps from input frames and use estimated depth values ​​in the depth map to filter items in the segmentation map that are beyond a threshold depth or range and / or are not connected to the segmentation target. The systems and techniques described herein can then generate a more accurate segmentation output. In some examples, the systems and techniques described herein can generate segmentation maps and depth maps from monocular images. The systems and techniques described herein can use the depth map to filter background items in the image and maintain the segmentation target of interest in the segmentation output.

[0028] 1 illustrates an exemplary image processing system 100, in accordance with some embodiments. Image processing system 100 can perform the segmentation techniques described herein. In addition, image processing system 100 can perform various image processing tasks and generate various image processing effects, as described herein. For example, image processing system 100 can perform image segmentation, foreground prediction, background replacement, depth of field effects, chromakeying effects, feature extraction, object detection, image recognition, machine vision, and / or any other image processing and computer vision tasks.

[0029] 1, image processing system 100 includes image capture device 102, storage 108, compute component 110, image processing engine 120, one or more neural network(s) 122, and rendering engine 124. Image processing system 100 may also optionally include one or more additional image capture devices 104 and one or more sensors 106, such as light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, accelerometers, gyroscopes, optical sensors, inertial measurement units (IMUs), proximity sensors, etc. In some cases, image processing system 100 may include multiple image capture devices capable of capturing images with different FOVs. For example, in a dual camera or image sensor application, the image processing system 100 may include image capture devices having different types of lenses (e.g., wide angle, telephoto, standard, zoom, etc.) capable of capturing images having different FOVs (e.g., different angles of view, different depths of field, etc.).

[0030] Image processing system 100 may be part of a computing device or multiple computing devices. In some embodiments, image processing system 100 may be part of electronic device(s), such as a camera system (e.g., digital camera, IP camera, video camera, security camera, etc.), a telephone system (e.g., smartphone, cellular phone, conferencing system, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a display device, a digital media player, a game console, a video streaming device, a drone, a computer in an automobile, an Internet-of-Things (IoT) device, a smart wearable device, an extended reality (XR) device (e.g., a head-mounted display, smart glasses, etc.), or any other suitable electronic device(s).

[0031] In some implementations, image capture device 102, image capture device 104, other sensor(s) 106, storage 108, compute component 110, image processing engine 120, neural network(s) 122, and rendering engine 124 may be part of the same computing device. For example, in some cases, image capture device 102, image capture device 104, other sensor(s) 106, storage 108, compute component 110, image processing engine 120, neural network(s) 122, and rendering engine 124 may be integrated into a smartphone, a laptop, a tablet computer, a smart wearable device, a gaming system, an XR device, and / or any other computing device. However, in some implementations, image capture device 102, image capture device 104, other sensor(s) 106, storage 108, compute component 110, image processing engine 120, neural network(s) 122, and / or rendering engine 124 may be part of two or more separate computing devices.

[0032] In some examples, image capture devices 102 and 104 may be any image and / or video capture device, such as a digital camera, a video camera, a smartphone camera, a camera device on an electronic device such as a television or computer, a camera system, etc. In some cases, image capture devices 102 and 104 may be part of a camera or computing device, such as a digital camera, a video camera, an IP camera, a smartphone, a smart television, a gaming system, etc. In some examples, image capture devices 102 and 104 may be part of a dual camera assembly. Image capture devices 102 and 104 may capture image and / or video content (e.g., raw image and / or video data), which may then be processed by compute component 110, image processing engine 120, neural network(s) 122, and / or rendering engine 124, as described herein.

[0033] In some cases, image capture devices 102 and 104 may include image sensors and / or lenses for capturing image data (e.g., still images, video frames, etc.). Image capture devices 102 and 104 may capture image data having different or the same FOV, including different or the same viewing angles, different or the same depth of field, different or the same size, etc. For example, in some cases, image capture devices 102 and 104 may include different image sensors with different FOV. In other examples, image capture devices 102 and 104 may include different types of lenses with different FOV, such as a wide-angle lens, a telephoto lens (e.g., short telephoto, medium telephoto, etc.), a standard lens, a zoom lens, etc. In some examples, image capture device 102 may include one type of lens and image capture device 104 may include a different type of lens. In some cases, image capture devices 102 and 104 may respond to different types of light. For example, in some cases, image capture device 102 may respond to visible light and image capture device 104 may respond to infrared light.

[0034] The other sensor(s) 106 may be any sensor for detecting and measuring information such as distance, motion, position, depth, speed, etc. Non-limiting examples of sensors include LIDAR, ultrasonic sensors, gyroscopes, accelerometers, magnetometers, RADAR, IMUs, audio sensors, light sensors, etc. In one exemplary embodiment, the sensor 106 may be a LIDAR configured to sense or measure distance and / or depth information that can be used in calculating depth of field and other effects. In some cases, the image processing system 100 may include other sensors, such as machine vision sensors, smart scene sensors, voice recognition sensors, shock sensors, position sensors, tilt sensors, light sensors, etc.

[0035] Storage 108 may include any storage device or devices for storing data, such as, for example, image data. Storage 108 may store data from any of the components of image processing system 100. For example, storage 108 may store data or measurements from image capture devices 102 and 104, other sensor(s) 106, any of compute components 110 (e.g., processing parameters, output, video, images, segmentation maps, depth maps, filtering results, calculation results, etc.), and / or any of image processing engine 120, neural network(s) 122, and / or rendering engine 124 (e.g., output images, processing results, parameters, etc.). In some embodiments, storage 108 may include a buffer for storing data (e.g., image data) for processing by compute components 110.

[0036] In some implementations, the compute component 110 may include a central processing unit (CPU) 112, a graphics processing unit (GPU) 114, a digital signal processor (DSP) 116, and / or an image signal processor (ISP) 118. The compute component 110 may perform various operations such as image enhancement, feature extraction, object or image segmentation, depth estimation, computer vision, graphics rendering, XR (e.g., augmented reality, virtual reality, mixed reality, etc.), image / video processing, sensor processing, recognition (e.g., text recognition, object recognition, feature recognition, face recognition, pattern recognition, scene recognition, etc.), foreground prediction, machine learning, filtering, depth of field effect calculation or rendering, tracking, localization, and / or any of the various operations described herein. In some examples, the compute component 110 may implement an image processing engine 120, a neural network(s) 122, and a rendering engine 124. In other examples, the compute component 110 may also implement one or more other processing engines.

[0037] The operations of image processing engine 120, neural network(s) 122, and rendering engine 124 may be performed by one or more of compute components 110. In an exemplary implementation, image processing engine 120 and neural network(s) 122 (and associated operations) may be implemented by CPU 112, DSP 116, and / or ISP 118, and rendering engine 124 (and associated operations) may be implemented by GPU 114. In some cases, compute components 110 may include other electronic circuitry or hardware, computer software, firmware, or any combination thereof, to perform any of the various operations described herein.

[0038] In some cases, the compute component 110 may receive data (e.g., image data, etc.) captured by the image capture devices 102 and / or 104 and process the data to generate an output image or video having certain visual and / or image processing effects, such as, for example, depth of field effects, background replacement, tracking, object detection, etc. For example, the compute component 110 may receive the image data (e.g., one or more still images or video frames, etc.) captured by the image capture devices 102 and 104, perform depth estimation, image segmentation, and depth filtering, and generate output segmentation results as described herein. The image (or frame) may be a red-green-blue (RGB) image having red, green, and blue color components per pixel, a luminance, red difference, blue difference (YCbCr) image having one luminance component and two chrominance (color) components (red chrominance and blue chrominance) per pixel, or any other suitable type of color or monochrome image.

[0039] The compute component 110 may implement the image processing engine 120 and neural network(s) 122 to perform various image processing operations and generate image effects. For example, the compute component 110 may implement the image processing engine 120 and neural network(s) 122 to perform feature extraction, superpixel detection, foreground prediction, spatial mapping, saliency detection, segmentation, depth estimation, depth filtering, pixel classification, cropping, upsampling / downsampling, blurring, modeling, filtering, color correction, noise reduction, scaling, ranking, adaptive Gaussian thresholding, and / or other image processing tasks. The compute component 110 may process image data captured by the image capture devices 102 and / or 104, image data in the storage 108, image data received from a remote source such as a remote camera, server, or content provider, image data obtained from a combination of sources, etc.

[0040] In some implementations, the compute component 110 may generate a depth map from a monocular image captured by the image capture device 102, generate a segmentation map from the monocular image, generate a refined or updated segmentation map based on depth filtering performed by comparing the depth map and the segmentation map to filter pixels / regions having at least a threshold depth, and generate a segmentation output. In some cases, the compute component 110 may use spatial information (e.g., a center prior map), a probability map, disparity information (e.g., a disparity map), an image query, a saliency map, etc. to segment objects and / or regions in one or more images and generate an output image with image effects such as depth of field effects. In other cases, the compute component 110 may use other information such as face detection information, sensor measurements (e.g., depth measurements), depth measurements, etc.

[0041] In some implementations, the compute component 110 may perform segmentation (e.g., foreground-background segmentation, object segmentation, etc.) with (or near) pixel-level or region-level accuracy. In some cases, the compute component 110 may perform segmentation using images with different FOVs. For example, the compute component 110 may perform segmentation using an image with a first FOV captured by the image capture device 102 and an image with a second FOV captured by the image capture device 104. Segmentation may also enable (or be used in conjunction with) other image adjustment or image processing operations such as, for example, but not limited to, depth extension and object recognition auto-exposure, auto-white balance, auto-focus, tone mapping, etc.

[0042] Although image processing system 100 is shown as including several components, one skilled in the art will appreciate that image processing system 100 may include more or fewer components than those shown in Figure 1. For example, image processing system 100 may also include one or more memory devices (e.g., RAM, ROM, cache, etc.), one or more network interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices, and / or other hardware or processing devices, in some cases not shown in Figure 1. Exemplary examples of computing devices and hardware components that may be implemented with image processing system 100 are described below with respect to Figure 8.

[0043] In some cases, semantic segmentation can generate a class mapping for each pixel of the view, and objects in the class, such as people, are identified from the image data. As mentioned above, in many cases, the accuracy of semantic segmentation can be reduced when there are more distant people or objects in the captured scene (e.g., relative to the foreground or target of interest, such as a person or object in the foreground). The reduced accuracy can be caused by the smaller size and / or smaller resolution of the people or objects that are more distant in the scene. Inaccuracies and / or inconsistencies in semantic segmentation can cause artifacts and / or flickering in the video when remote people and / or objects (e.g., more distant people and / or objects in the captured scene and / or background) are included in the captured scene. Accurate semantic segmentation can detect and segment people and / or objects in the background that are not of interest, as well as the target(s) of interest, such as foreground objects.

[0044] FIG. 2 illustrates an exemplary scene 200 with multiple objects 210 in the background. In this example, a person 202 in the scene is a target of interest for semantic segmentation. The person 202 has been detected in the scene by the image processing system 100. However, an object 210 that is not a target of interest for segmentation has also been detected. As shown, the object 210 is farther away from the person 202 and is smaller than the person 202, and therefore more difficult to distinguish, filter, and / or determine that such an object is not of interest. This may result in inaccuracies / inconsistencies in the segmentation. Furthermore, this may cause flickering in the video of the scene 200. For example, when the image processing system 100 performs semantic segmentation of frames capturing the scene 200, the object 210 may be detected in some frames and not detected in other frames. This may cause flickering between frames as the object 210 is segmented in some frames and not segmented in other frames.

[0045] FIG. 3 illustrates an example process 300 for segmentation using depth estimation, according to some embodiments of the present disclosure. The process 300 can use depth estimation in addition to semantic segmentation to improve the stability of the segmentation results. For example, the process 300 can reduce or avoid flickering as described above, resulting in more accurate segmentation results, etc. In some embodiments, the process 300 can use monocular depth estimation to filter certain portions of the segmentation results. For example, the process 300 can use monocular depth estimation to filter objects and / or people in the segmentation map that are farther away (e.g., having at least a threshold depth) from a target of interest (e.g., a foreground target, etc.) and generate a segmentation result using depth filtering.

[0046] 3, process 300 generates (e.g., via image processing system 100) a segmentation map 304 from an input frame 302. In some examples, process 300 may perform semantic segmentation on the input frame 302 to generate the segmentation map 304. In addition, process 300 generates a depth estimate 306 from the input frame 302. In some examples, the depth estimate 306 may include a monocular depth estimate, and the input frame 302 may include a monocular camera image frame.

[0047] In some cases, the depth estimation 306 may include a depth map of the input frame 302. In some implementations, the depth estimation 306 may estimate the depth of every pixel of the input frame 302. The process 300 may use the depth estimation 306 to perform depth filtering 308. For example, the depth estimation 306 may be used to filter out unwanted items in the background (e.g., smaller / distant objects, etc.) to minimize or prevent flicker. For example, the process 300 may compare the segmentation map 304 to the depth estimation 306. The process 300 may match salient depth regions from the depth estimation 306 with a predicted mask in the segmentation map 304. The process 300 may retain any predicted masks in the segmentation map 304 that match and / or at least partially overlap one or more salient depth regions from the depth estimate 306 and filter out any predicted masks in the segmentation map 304 that do not match and / or at least partially overlap one or more salient depth regions from the depth estimate 306.

[0048] Based on the depth filtering 308, the process 300 may output a segmentation result 310. The segmentation result 310 may exclude or filter any predicted masks in the segmentation map 304 that do not match and / or at least partially overlap with one or more salient depth regions from the depth estimate 306. Thus, in some examples, the segmentation result 310 may include a filtered segmentation map. For example, the segmentation result 310 may retain any predicted masks in the segmentation map 304 that match and / or overlap with one or more salient depth regions from the depth estimate 306. In some examples, the items removed / filtered from the segmentation map 304 may include items (e.g., objects, people, regions, etc.) in the depth map (e.g., depth estimate 306) that have greater depth values ​​than one or more items in the depth map that correspond to one or more segmentation masks or items in the segmentation map 304.

[0049] 4 illustrates an example depth filtering process 400 for generating a segmented output based on a segmentation map and estimated depth information. In some embodiments, the depth filtering process 400 may include, represent, or be the same as the depth filtering 308 shown in FIG.

[0050] In this embodiment, the depth filtering system 410 receives the segmentation map 402 and the depth map 404. In some embodiments, the depth filtering system 410 may be implemented by the image processing system 100. The segmentation map 402 and the depth map 404 may be based on input frames, as previously described. For example, the segmentation map 402 and the depth map 404 may be based on monocular camera frames.

[0051] At block 412, the depth filtering system 410 may apply adaptive Gaussian thresholding to the depth map 404. In some implementations, adaptive Gaussian thresholding may help identify targets of interest in the depth map 404 based on various depth values ​​in the depth map 404. Additionally, adaptive Gaussian thresholding may be used to select frame regions having depth values ​​that differ from the depth values ​​of surrounding / background pixels by a threshold amount. For example, in some cases, adaptive Gaussian thresholding may identify one or more depth values ​​of targets of interest in the depth map 404 and set a depth threshold or range that is used to subtract regions / pixels / objects in the depth map 404 that do not correspond to and / or are not connected to the targets of interest in the depth map 404. For example, adaptive Gaussian thresholding may select / keep target region(s) having a particular depth value(s) and exclude / subtract any pixels / regions in the depth map 404 that are above the depth threshold or range and / or are not connected to the selected target region(s).

[0052] In some embodiments, the depth filtering system 410 may model the background of a scene (e.g., captured in an input frame) using any suitable background subtraction technique (also referred to as background extraction). For example, in some cases, the depth filtering system 410 may use a Gaussian distribution model for each pixel location in the depth map 404, with parameters of mean and variance to model each pixel location. In some embodiments, the value of a previous pixel at a particular pixel location may be used to calculate the mean and variance of a target Gaussian model for that pixel location. When a pixel at a given location in the input frame is processed, its value may be evaluated by the current Gaussian distribution for this pixel location. Classification of a pixel as either a foreground pixel or a background pixel may be performed by comparing the difference between the pixel value and the mean of a specified Gaussian model. In one exemplary embodiment, if the distance between the pixel value and the Gaussian mean is less than a certain amount of variance, the pixel may be classified as a background pixel. Otherwise, in this exemplary embodiment, the pixel may be classified as a foreground pixel.

[0053] At block 414, the depth filtering system 410 may perform noise reduction on the depth map resulting from the adaptive Gaussian thresholding. In some implementations, the depth filtering system 410 may perform noise reduction via erosion and dilation operations. For example, in some cases, the depth filtering system 410 may perform a morphological function to filter foreground pixels in the depth map 404. The morphological function may include erosion and dilation functions. In one example, an erosion function may be applied, followed by a series of one or more dilation functions. The erosion function may be applied to remove pixels on the target (e.g., object / region) boundary.

[0054] For example, the depth filtering system 410 may apply an erosion function to a filter window of a central pixel being processed. The window may be applied to each foreground pixel (as the central pixel) in the foreground mask. The erosion function may include an erosion operation that sets a current foreground pixel in the foreground mask (acting as the central pixel) to a background pixel if one or more of its neighboring pixels in the window are background pixels. Such an erosion operation may be referred to as a strong erosion operation or a single-neighborhood erosion operation, where the neighboring pixels of the current central pixel include the pixels in the window, and the additional pixel is the current central pixel.

[0055] The dilation operation may be used to enhance the boundaries of foreground objects. For example, the depth filtering system 410 may apply a dilation function to a filter window of a center pixel. The dilation window may be applied to each background pixel (as the center pixel) in the foreground mask. The dilation function may include a dilation operation that sets a current background pixel in the foreground mask (acting as the center pixel) as a foreground pixel if one or more of its neighboring pixels in the window are foreground pixels. The neighboring pixels of the current center pixel include the pixels in the window, and the additional pixel is the current center pixel. In some examples, multiple dilation operations may be applied after the erosion function is applied. In one exemplary embodiment, multiple function calls of a dilation of a particular window size may be applied to the foreground mask. In some embodiments, an erosion function may be applied first to remove pixels that are noise, and a series of dilation functions may be applied to refine the foreground pixels. In one exemplary embodiment, an erosion function with a particular window size is called first, and multiple function calls of a dilation of a particular window size are applied to the foreground mask.

[0056] In some cases, after morphological operations are performed, the depth filtering system 410 may apply connected component analysis to connect neighboring foreground pixels and organize connected components and blobs. In some implementations of connected component analysis, one or more bounding boxes are returned in such a way that each bounding box contains one component of the connected pixels.

[0057] At block 416, the depth filtering system 410 may perform intersection over union (IOU) matching between the segmentation map 402 and the depth map 404 after adaptive Gaussian thresholding and noise reduction. The IOU matching may match salient depth regions in the depth map with predicted masks from the segmentation map 402 based on their IOU. In some implementations, the IOU may measure the overlap between a depth mask (e.g., salient depth regions) or a bounding shape (e.g., a bounding box, etc.) in the depth map and a segmentation mask or a bounding shape in the segmentation map 402.

[0058] At block 418, the depth filtering system 410 may perform mask filtering based on the IOU matching. For example, the depth filtering system 410 may subtract / filter any masks (or boundary shapes) in the segmentation map 402 that have an IOU score below a threshold (e.g., do not have sufficient overlap with the depth mask(s) in the depth map).

[0059] The depth filtering system 410 may then generate a segmentation output 420 that includes the segmentation map 402 that does not include masks (or boundary shapes) with an IOU score below a threshold. The segmentation output 420 may provide greater segmentation accuracy / stability and prevent or minimize flicker in a sequence of frames associated with an input frame.

[0060] 5 illustrates an example training phase 500 and inference phase 520 for segmentation with depth filtering, according to some embodiments of the present disclosure. In the training phase 500, the image processing system 100 may obtain input frames 502 and perform segmentation 504 to generate a segmentation map. The image processing system 100 may also perform depth estimation 506 on the input frames 502 to generate a depth map. In some embodiments, the input frames may include monocular camera frames, and the depth map may include a monocular depth estimation.

[0061] The image processing system 100 can use the segmentation map from segmentation 504 to perform supervised segmentation learning 508. In some embodiments, the image processing system 100 can implement a neural network (e.g., neural network 122) to perform the segmentation in the training phase 500 and inference phase 520. In some cases, in the supervised segmentation learning 508 in the training phase 500, the image processing system 100 can use a training data set to help calculate a loss for the output from segmentation 504. The image processing system 100 can adjust weights in the neural network based on the calculated loss to improve its segmentation results.

[0062] In some embodiments, the image processing system 100 may implement a neural network (e.g., neural network 122) to perform depth estimation in the training phase 500 and the inference phase 520. In the training phase 500, the image processing system 100 may perform self-supervised depth learning 510 using the output from the depth estimation 506. In some embodiments, the image processing system 100 may use a dataset of target outputs to generate a depth estimation model. In some cases, the image processing system 100 may use the depth estimation model to calculate depth estimates and / or determine depth estimation losses. In some embodiments, the image processing system 100 may calculate depth estimates and determine whether they match associated frames. The image processing system 100 may then adjust the weights of the neural network based on the matching results and / or the calculated losses.

[0063] During the inference stage 520, the image processing system 100 may perform the process 300 and the depth filtering process 400 as described above with respect to Figures 3 and 4. For example, the image processing system 100 may perform semantic segmentation 524 on the input frames 522 to generate a segmentation map. The image processing system 100 may also perform depth estimation 526 on the input frames 522 to generate a depth map.

[0064] The image processing system 100 can then perform depth filtering 528 using the segmentation map and the depth map. The depth filtering 528 can compare the segmentation map and the depth map to subtract regions / pixels that do not have a threshold amount of overlap between the segmentation map and the depth map. For example, as previously described, the image processing system 100 can calculate an IOU score between the segmentation map and the depth map and subtract pixels / regions that have an IOU score below a threshold. The image processing system 100 can generate a segmentation output 530 (e.g., a filtered segmentation map) based on the depth filtering 528. The segmentation output 530 can provide higher segmentation accuracy / stability and prevent or minimize flicker in the sequence of frames associated with the input frame 522.

[0065] As shown, the image processing system 100 can use three-dimensional (3D) depth prediction of the current frame to filter out objects that are part of the background, unwanted, distant, small, and / or combinations thereof. Segmentation using depth filtering as described herein can produce reliable temporal consistency of segmented frames.

[0066] 6 is a diagram illustrating an example of segmented frames without and with depth filtering, where an input frame 602 is used to generate a segmented frame 604 that does not include depth filtering. As shown, the segmented frame 604 detects (e.g., segmented, masked, identified) targets 612 of interest in the foreground, but also detects various objects 610 in the background that are not of interest. In a sequence of frames that includes the segmented frame 604, the detected objects 610 may cause flickering because they are detected in the segmented frame 604 and not in other frames of the sequence of frames.

[0067] Meanwhile, Fig. 6 also illustrates a segmented frame 608 using depth filtering as described herein. The segmented frame 608 is generated based on the estimated depth 606 calculated for the input frame 602 and the segmentation map calculated for the input frame. As shown, since the object 610 has been filtered using the estimated depth 606, the segmented frame 608 successfully detects the target of interest 612 without also detecting the object 610. As a result, the segmented frame 608 does not cause flicker from the object 610 being detected in some frames and not in others.

[0068] 7 is a flow chart of an example of a process 700 for semantic segmentation with depth filtering, according to some embodiments of the disclosure. At block 702, the process 700 may include obtaining a frame capturing a scene. The frame may include one or more foreground regions and one or more background regions. In some embodiments, the frame is a monocular frame captured by a monocular camera device (e.g., image capture device 102).

[0069] At block 704, the process 700 may include generating a first segmentation map (e.g., segmentation map 304, segmentation map 402) based on the frame, the first segmentation map including a target segmentation mask that identifies a target of interest (e.g., person 202 or target of interest 612) and one or more background masks that identify one or more background regions of the frame (e.g., object 210 or subject 610).

[0070] At block 706, the process 700 may include generating a second segmentation map (e.g., segmentation result 310, segmentation output 420) that includes the first segmentation map with the one or more background masks filtered. In some embodiments, the one or more background masks may be filtered from the first segmentation map based on a depth map (e.g., depth estimate 306, depth map 404) associated with the frame.

[0071] In some aspects, the process 700 may include generating a depth map (e.g., depth estimate 306, depth map 404) based on the frame, where the depth map includes depth values ​​associated with pixels of the frame.

[0072] In some aspects, the process 700 can include filtering one or more background masks from the first segmentation map based on depth values ​​in the depth map. In some examples, generating the second segmentation map can include determining a threshold difference between respective depth values ​​associated with the one or more background masks and respective depth values ​​associated with a target segmentation mask that identifies a target of interest based on a comparison of the first segmentation map and the depth map. In some aspects, the process 700 can include removing one or more background masks from the first segmentation map based on a threshold difference between respective depth values ​​associated with the one or more background masks and respective depth values ​​associated with a target segmentation mask that identifies a target of interest.

[0073] In some cases, the depth map includes a set of depth masks associated with depth values ​​corresponding to pixels of the frame. In some examples, generating the second segmentation map can include determining an overlap between a target segmentation mask identifying a target of interest and one or more depth masks from the set of depth masks in the depth map based on a comparison of the first segmentation map and the depth map, retaining the target segmentation mask identifying the target of interest based on the overlap, and filtering the one or more background masks from the first segmentation map based on additional overlap between the one or more background masks and the one or more additional depth masks from the set of depth masks.

[0074] In some examples, generating the second segmentation map may further include determining that a difference between a depth value associated with the one or more additional depth masks and a depth value associated with the one or more depth masks exceeds a threshold, and filtering one or more background masks from the first segmentation map based on the difference exceeding the threshold. In some cases, the one or more depth masks correspond to a target of interest and the one or more additional depth masks correspond to one or more background regions of the frame.

[0075] In some cases, generating the second segmentation map may include determining an intersection over union (IOU) score associated with a depth region from the depth map and a predicted mask from the first segmentation map; matching the depth region from the depth map with a predicted mask from the first segmentation map based on the IOU score, the predicted mask including a target segmentation mask identifying a target of interest and one or more background masks identifying one or more background regions of the frame; and filtering the one or more background masks from the first segmentation map based on a determination that one or more IOU scores associated with the one or more background masks are below a threshold.

[0076] In some aspects, the process 700 may include generating a modified frame based on the frame and the second segmentation map. In some implementations, the modified frame may include at least one of a visual effect, an extended reality effect, an image processing effect, a blur effect, an image recognition effect, an object detection effect, a computer graphics effect, a chromakeying effect, and an image stylization effect.

[0077] In some aspects, the process 700 may include applying adaptive Gaussian thresholding and noise reduction to the depth map prior to filtering one or more background masks from the first segmentation map.

[0078] In some embodiments, the first segmentation map and the second segmentation map are generated using one or more neural networks. In some embodiments, the depth map is generated using a neural network.

[0079] In some examples, processes 300, 400, and / or 700 may be performed by one or more computing devices or apparatuses. In one exemplary example, processes 300, 400, and / or 700 may be performed by one or more computing devices having image processing system 100 shown in FIG. 1 and / or computing device architecture 800 shown in FIG. 8. In some cases, such computing devices or apparatuses may include a processor, microprocessor, microcomputer, or other components of a device configured to perform steps of processes 300, 400, and / or 700. In some examples, such computing devices or apparatuses may include one or more sensors configured to capture image data. For example, the computing device may include a smartphone, a head mounted display, a mobile device, a camera, a tablet computer, or other suitable device. In some examples, such computing devices or apparatuses may include a camera configured to capture one or more images or videos. In some cases, such computing devices may include a display for displaying images. In some examples, one or more sensors and / or cameras are separate from the computing device, in which case the computing device receives the sensed data. Such a computing device may further include a network interface configured to communicate data.

[0080] The components of a computing device may be implemented in circuits. For example, the components may include and / or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform various operations described herein. The computing device may further include a display (as an example of an output device or in addition to an output device), a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0081] Processes 300, 400, and 700 are illustrated as logic flow diagrams, whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement a process.

[0082] Additionally, processes 300, 400, and / or 700 may be executed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a number of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0083] 8 illustrates an exemplary computing device architecture 800 of an exemplary computing device that may implement various techniques described herein. For example, the computing device architecture 800 may implement at least some portions of the image processing system 100 illustrated in FIG. 1. The components of the computing device architecture 800 are shown in electrical communication with each other using a connection 805, such as a bus. The exemplary computing device architecture 800 includes a processing unit (CPU or processor) 810 and a computing device connection 805 that couples various computing device components to the processor 810, including a computing device memory 815, such as a read only memory (ROM) 820 and a random access memory (RAM) 825.

[0084] The computing device architecture 800 may include a cache of high-speed memory directly connected to, near, or integrated as part of the processor 810. The computing device architecture 800 may copy data from the memory 815 and / or storage device 830 to the cache 812 for faster access by the processor 810. In this manner, the cache may provide a performance boost that avoids delays to the processor 810 while waiting for data. These and other modules may control or be configured to control the processor 810 to perform various actions. Other computing device memories 815 may be available for use as well. The memory 815 may include multiple different types of memory having different performance characteristics.

[0085] Processor 810 may include any general purpose processor, as well as hardware or software services, such as service 1 832, service 2 834, and service 3 836 stored in storage device 830, configured to control processor 810, and special purpose processors where software instructions are built into the processor design. Processor 810 may be a self-contained system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0086] To enable user interaction with the computing device architecture 800, the input device 845 can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. The output device 835 can also be one or more of several output mechanisms known to those skilled in the art, such as a display, projector, television, speaker device, etc. In some cases, a multimodal computing device may enable a user to provide multiple types of input to communicate with the computing device architecture 800. The communication interface 840 can generally govern and manage user input and computing device output. There is no constraint to operate on any particular hardware configuration, and therefore the basic functions herein may be easily replaced with improved hardware or firmware configurations as they are developed.

[0087] The storage device 830 is a non-volatile memory and may be a hard disk or other type of computer readable medium capable of storing data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, random access memories (RAMs) 185, a read only memory (ROM) 820, and hybrids thereof. The storage device 830 may include services 832, 834, 836 for controlling the processor 810. Other hardware or software modules are contemplated. The storage device 830 may be connected to a computing device connection 805. In one aspect, a hardware module that performs a particular function may include software components stored in a computer readable medium that connect with necessary hardware components, such as the processor 810, the connection 805, the output device 835, etc., to perform the function.

[0088] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, store, or convey instruction(s) and / or data. Computer-readable media may include non-transitory media on which data is stored and does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memories, or memory devices. Computer-readable media may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0089] In some embodiments, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when mentioned, non-transitory computer readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0090] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For clarity of explanation, in some cases, the technology may be presented as including individual functional blocks comprising devices, device components, and steps or routines in a method embodied in software or a combination of hardware and software. Additional components may be used other than those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0091] Individual embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowcharts may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagrams. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0092] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored on or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a processing device to perform a function or group of functions, or in some cases configure a general-purpose computer, a special-purpose computer, or a processing device to perform a function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, network-attached storage devices, etc.

[0093] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) to perform the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor or processors may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small-footprint personal computers, personal digital assistants, rack-mounted devices, standalone devices, and the like. The functionality described herein may also be embodied in a peripheral device or an add-in card. Such functionality may also be implemented on a circuit board among different chips, or on different processes executing in a single device, as further examples.

[0094] The instructions, media for carrying such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0095] In the above description, aspects of the present application are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, while exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in various other ways, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. The various features and aspects of the present application described above may be used individually or jointly. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative and not restrictive. For purposes of illustration, the methods have been described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in an order different from that described.

[0096] Those skilled in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.

[0097] When a component is described as being "configured to" perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0098] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component and / or that is in communication, either directly or indirectly, with another component (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0099] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, a claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. As another example, a claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can additionally include unrecited items in the set of A and B.

[0100] The various exemplary logic blocks, modules, circuits, and algorithm steps described with respect to the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, and such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0101] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium including program code including instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read only memory (EEPROM), flash memory, magnetic or optical data storage medium, etc. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, which may carry or communicate program code in the form of instructions or data structures and which may be accessed, read, and / or executed by a computer.

[0102] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but the processor may alternatively be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0103] Illustrative aspects of the present disclosure include the following.

[0104] Aspect 1. An apparatus for image segmentation comprising: a memory; and one or more processors coupled to the memory, wherein the one or more processors are configured to: acquire frames capturing a scene; generate a first segmentation map based on the frames, the first segmentation map including a target segmentation mask that identifies a target of interest and one or more background masks that identify one or more background regions of the frames; and generate a second segmentation map including the first segmentation map with the one or more background masks filtered, where the one or more background masks are filtered from the first segmentation map based on a depth map associated with the frames.

[0105] Aspect 2. The apparatus of aspect 1, wherein to generate the second segmentation map, the one or more processors are configured to determine a threshold difference between respective depth values ​​associated with one or more background masks and respective depth values ​​associated with a target segmentation mask that identifies a target of interest based on a comparison of the first segmentation map and the depth map.

[0106] Aspect 3. The apparatus of aspect 2, wherein to generate the second segmentation map, the one or more processors are configured to remove one or more background masks from the first segmentation map based on a threshold difference between respective depth values ​​associated with the one or more background masks and respective depth values ​​associated with a target segmentation mask that identifies a target of interest.

[0107] Aspect 4. The apparatus of any one of aspects 1 to 3, wherein the depth map includes a set of depth masks associated with depth values ​​corresponding to pixels of the frame, and to generate the second segmentation map, the one or more processors are configured to: determine an overlap between a target segmentation mask identifying a target of interest and one or more depth masks from the set of depth masks in the depth map based on a comparison of the first segmentation map and the depth map, retain the target segmentation mask identifying the target of interest based on the overlap, and filter one or more background masks from the first segmentation map based on additional overlap between the one or more background masks and one or more additional depth masks from the set of depth masks.

[0108] Aspect 5. The apparatus of aspect 4, wherein to generate a second segmentation map, the one or more processors are configured to determine that a difference between depth values ​​associated with the one or more additional depth masks and depth values ​​associated with the one or more depth masks exceeds a threshold, and filter one or more background masks from the first segmentation map based on the difference exceeding the threshold, wherein the one or more depth masks correspond to a target of interest and the one or more additional depth masks correspond to one or more background regions of the frame.

[0109] Aspect 6. The apparatus of any one of aspects 1 to 5, wherein to generate the second segmentation map, the one or more processors are configured to: determine intersection over union (IOU) scores associated with depth regions from the depth map and predicted masks from the first segmentation map; match the depth regions from the depth map against predicted masks from the first segmentation map based on the IOU scores, the predicted masks including a target segmentation mask identifying a target of interest and one or more background masks identifying one or more background regions of the frame; and filter the one or more background masks from the first segmentation map based on a determination that one or more IOU scores associated with the one or more background masks are less than a threshold.

[0110] Aspect 7. The apparatus of aspect 6, wherein the one or more processors are configured to apply adaptive Gaussian thresholding and noise reduction to the depth map prior to filtering one or more background masks from the first segmentation map.

[0111] Example 8. An apparatus as described in any one of Examples 1 to 7, wherein the frames comprise monocular frames generated by a monocular image capture device.

[0112] Aspect 9. An apparatus described in any one of aspects 1 to 8, wherein the first segmentation map and the second segmentation map are generated using one or more neural networks.

[0113] Embodiment 10. An apparatus as described in any one of embodiments 1 to 9, wherein the one or more processors are configured to generate the depth map using a neural network.

[0114] Example 11. An apparatus according to any one of Examples 1 to 10, wherein the one or more processors are configured to generate a modified frame based on the frame and the second segmentation map.

[0115] Aspect 12. The apparatus of aspect 11, wherein the modified frame includes at least one of a visual effect, an extended reality effect, an image processing effect, a blur effect, an image recognition effect, an object detection effect, a computer graphics effect, a chroma keying effect, and an image stylization effect.

[0116] Aspect 13. The apparatus of any one of aspects 1 to 12, further comprising an image capture device, wherein the frames are generated by the image capture device.

[0117] Example 14. The apparatus of any one of examples 1 to 13, wherein the apparatus comprises a mobile device.

[0118] Aspect 15. A method of image segmentation comprising: acquiring a frame capturing a scene; generating a first segmentation map based on the frame, the first segmentation map including a target segmentation mask identifying a target of interest and one or more background masks identifying one or more background regions of the frame; and generating a second segmentation map including the first segmentation map with the one or more background masks filtered, where the one or more background masks are filtered from the first segmentation map based on a depth map associated with the frame.

[0119] Aspect 16. The method of aspect 15, wherein generating a second segmentation map includes determining a threshold difference between respective depth values ​​associated with one or more background masks and respective depth values ​​associated with a target segmentation mask that identifies a target of interest based on a comparison of the first segmentation map and the depth map.

[0120] Aspect 17. The method of aspect 16, wherein generating the second segmentation map further includes removing one or more background masks from the first segmentation map based on a threshold difference between respective depth values ​​associated with the one or more background masks and respective depth values ​​associated with a target segmentation mask that identifies the target of interest.

[0121] Aspect 18. The method of any one of aspects 15 to 17, wherein the depth map includes a set of depth masks associated with depth values ​​corresponding to pixels of the frame, and generating the second segmentation map includes: determining an overlap between a target segmentation mask identifying a target of interest and one or more depth masks from the set of depth masks in the depth map based on a comparison between the first segmentation map and the depth map; retaining the target segmentation mask identifying the target of interest based on the overlap; and filtering one or more background masks from the first segmentation map based on additional overlap between the one or more background masks and one or more additional depth masks from the set of depth masks.

[0122] Aspect 19. The method of aspect 18, wherein generating a second segmentation map further includes determining that a difference between depth values ​​associated with the one or more additional depth masks and depth values ​​associated with the one or more depth masks exceeds a threshold, and filtering one or more background masks from the first segmentation map based on the difference exceeding the threshold, wherein the one or more depth masks correspond to a target of interest and the one or more additional depth masks correspond to one or more background regions of the frame.

[0123] Aspect 20. The method of any one of aspects 15 to 19, wherein generating the second segmentation map comprises determining an intersection over union (IOU) score associated with a depth region from the depth map and a predicted mask from the first segmentation map; matching the depth region from the depth map with a predicted mask from the first segmentation map based on the IOU score, the predicted mask including a target segmentation mask identifying a target of interest and one or more background masks identifying one or more background regions of the frame; and filtering the one or more background masks from the first segmentation map based on a determination that one or more IOU scores associated with the one or more background masks are below a threshold.

[0124] Aspect 21. The method of aspect 20, wherein one or more processors are configured to apply adaptive Gaussian thresholding and noise reduction to the depth map before filtering one or more background masks from the first segmentation map.

[0125] Example 22. The method of any one of examples 15 to 21, wherein the frames comprise monocular frames generated by a monocular image capture device.

[0126] Embodiment 23. The method of any one of embodiments 15 to 22, wherein the first segmentation map and the second segmentation map are generated using one or more neural networks.

[0127] Embodiment 24. The method of any one of embodiments 15 to 23, further comprising generating the depth map using a neural network.

[0128] Example 25. The method of any one of examples 15 to 24, further comprising generating a modified frame based on the frame and the second segmentation map.

[0129] Aspect 26. The method of aspect 25, wherein the modified frame includes at least one of a visual effect, an extended reality effect, an image processing effect, a blur effect, an image recognition effect, an object detection effect, a computer graphics effect, a chroma keying effect, and an image stylization effect.

[0130] Aspect 27. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of aspects 15 to 26.

[0131] Example 28. An apparatus comprising means for performing the method according to any one of examples 15 to 26.

Claims

1. 1. An apparatus for image segmentation, comprising: Memory, one or more processors coupled to the memory; wherein the one or more processors: Get a frame capturing the scene, generating a first segmentation map based on the frame, the first segmentation map including a target segmentation mask that identifies a target of interest and one or more background masks that identify one or more background regions of the frame; generating a second segmentation map that includes the first segmentation map with the one or more background masks filtered, the one or more background masks being filtered from the first segmentation map based on a depth map associated with the frame; It is configured as follows: Device.

2. To generate the second segmentation map, the one or more processors: determining a threshold difference between each depth value associated with the one or more background masks and each depth value associated with the target segmentation mask that identifies the target of the object based on a comparison of the first segmentation map and the depth map. It is configured as follows:

2. The apparatus of claim 1.

3. To generate the second segmentation map, the one or more processors: removing the one or more background masks from the first segmentation map based on the threshold difference between respective depth values ​​associated with the one or more background masks and respective depth values ​​associated with the target segmentation mask that identify the target of interest. It is configured as follows:

3. The apparatus of claim 2.

4. The depth map includes a set of depth masks associated with depth values ​​corresponding to pixels of the frame, and to generate the second segmentation map, the one or more processors: determining an overlap between the target segmentation mask identifying the target of interest and one or more depth masks from the set of depth masks in the depth map based on a comparison of the first segmentation map and the depth map; maintain the target segmentation mask that identifies the target of interest based on the overlap; filtering the one or more background masks from the first segmentation map based on an additional overlap between the one or more background masks and one or more additional depth masks from the set of depth masks. It is configured as follows:

2. The apparatus of claim 1.

5. To generate the second segmentation map, the one or more processors: determining that a difference between a depth value associated with the one or more additional depth masks and a depth value associated with the one or more depth masks exceeds a threshold; filtering the one or more background masks from the first segmentation map based on the difference exceeding the threshold, wherein the one or more depth masks correspond to the target of interest and the one or more additional depth masks correspond to the one or more background regions of the frame.

5. The apparatus of claim 4.

6. To generate the second segmentation map, the one or more processors: determining an intersection over union (IOU) score associated with a depth region from the depth map and a predicted mask from the first segmentation map; matching the depth regions from the depth map against the predicted masks from the first segmentation map, the predicted masks including the target segmentation mask identifying the target of interest and the one or more background masks identifying the one or more background regions of the frame based on the IOU score; filtering the one or more background masks from the first segmentation map based on a determination that one or more IOU scores associated with the one or more background masks are less than a threshold. It is configured as follows:

2. The apparatus of claim 1.

7. the one or more processors: configured to apply adaptive Gaussian thresholding and noise reduction to the depth map prior to filtering the one or more background masks from the first segmentation map.

7. The apparatus of claim 6.

8. The apparatus of claim 1 , wherein the frames comprise monocular frames generated by a monocular image capture device.

9. The apparatus of claim 1 , wherein the first segmentation map and the second segmentation map are generated using one or more neural networks.

10. The apparatus of claim 1 , wherein the one or more processors are configured to generate the depth map using a neural network.

11. the one or more processors: generating a modified frame based on the frame and the second segmentation map.

2. The apparatus of claim 1.

12. 12. The apparatus of claim 11, wherein the modified frame comprises at least one of a visual effect, an extended reality effect, an image processing effect, a blur effect, an image recognition effect, an object detection effect, a computer graphics effect, a chroma keying effect, and an image stylization effect.

13. The apparatus of claim 1 , further comprising an image capture device, the frames being generated by the image capture device.

14. The apparatus of claim 1 , wherein the apparatus comprises a mobile device.

15. 1. A method of image segmentation comprising the steps of: obtaining a frame capturing the scene; generating a first segmentation map based on the frame, the first segmentation map including a target segmentation mask identifying a target of interest and one or more background masks identifying one or more background regions of the frame; generating a second segmentation map that includes the first segmentation map with the one or more background masks filtered, the one or more background masks being filtered from the first segmentation map based on a depth map associated with the frame; A method comprising:

16. Generating the second segmentation map includes: determining a threshold difference between respective depth values ​​associated with the one or more background masks and respective depth values ​​associated with the target segmentation masks that identify the target of interest based on a comparison of the first segmentation map and the depth map; Including, The method of claim 15.

17. Generating the second segmentation map includes: removing the one or more background masks from the first segmentation map based on the threshold difference between respective depth values ​​associated with the one or more background masks and respective depth values ​​associated with the target segmentation mask that identify the target of interest; Further comprising:

17. The method of claim 16.

18. The depth map includes a set of depth masks associated with depth values ​​corresponding to pixels of the frame, and generating the second segmentation map includes: determining an overlap between the target segmentation mask identifying the target of interest and one or more depth masks from the set of depth masks in the depth map based on a comparison of the first segmentation map and the depth map; maintaining the target segmentation mask that identifies the target of interest based on the overlap; filtering the one or more background masks from the first segmentation map based on an additional overlap between the one or more background masks and one or more additional depth masks from the set of depth masks; Including, The method of claim 15.

19. Generating the second segmentation map includes: determining that a difference between a depth value associated with the one or more additional depth masks and a depth value associated with the one or more depth masks exceeds a threshold; filtering the one or more background masks from the first segmentation map based on the difference exceeding the threshold; and wherein the one or more depth masks correspond to the target of interest and the one or more additional depth masks correspond to the one or more background regions of the frame.

20. The method of claim 18.

20. Generating the second segmentation map includes: determining an intersection over union (IOU) score associated with a depth region from the depth map and a predicted mask from the first segmentation map; matching the depth regions from the depth map based on the IOU score against the predicted masks from the first segmentation map, the predicted masks including the target segmentation mask identifying the target of interest and the one or more background masks identifying the one or more background regions of the frame; filtering the one or more background masks from the first segmentation map based on a determination that one or more IOU scores associated with the one or more background masks are less than a threshold; Including, The method of claim 15.

21. The method of claim 20 , further comprising applying adaptive Gaussian thresholding and noise reduction to the depth map prior to filtering the one or more background masks from the first segmentation map.

22. The method of claim 15 , wherein the frames comprise monocular frames generated by a monocular image capture device.

23. The method of claim 15 , wherein the first segmentation map and the second segmentation map are generated using one or more neural networks.

24. The method of claim 15 , further comprising generating the depth map using a neural network.

25. The method of claim 15 , further comprising generating a modified frame based on the frame and the second segmentation map.

26. 26. The method of claim 25, wherein the modified frame comprises at least one of a visual effect, an extended reality effect, an image processing effect, a blur effect, an image recognition effect, an object detection effect, a computer graphics effect, a chroma keying effect, and an image stylization effect.

27. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: Get a frame that captures the scene, generating a first segmentation map based on the frame, the first segmentation map including a target segmentation mask identifying a target of interest and one or more background masks identifying one or more background regions of the frame; generating a second segmentation map that includes the first segmentation map with the one or more background masks filtered, the one or more background masks being filtered from the first segmentation map based on a depth map associated with the frame; Non-transitory computer-readable medium.

28. Generating the second segmentation map includes: determining a threshold difference between respective depth values ​​associated with the one or more background masks and respective depth values ​​associated with the target segmentation masks that identify the target of interest based on a comparison of the first segmentation map and the depth map; Including, 28. The non-transitory computer readable medium of claim 27.

29. Generating the second segmentation map includes: removing the one or more background masks from the first segmentation map based on the threshold difference between respective depth values ​​associated with the one or more background masks and respective depth values ​​associated with the target segmentation mask that identify the target of interest; Further comprising:

30. The non-transitory computer readable medium of claim 28.

Citation Information

Patent Citations

  • Monocular depth estimation method, device, equipment, and storage medium

    JP2021515939A

  • Self-supervised training of depth estimation models using depth hints

    JP2021522603A