Method and apparatus for performing depth estimation of an object

Abstract

A method and apparatus for performing depth estimation of an object in an image of a scene is provided. The method includes: capturing an image of the scene; obtaining pixel intensity data and event data from the image by a sensor; generating an event depth map using the event data, wherein the event data includes event map data of the image and event velocity data of the image; and generating a depth map of the object in the image using the event depth map and the pixel intensity data.

Description

Technical Field

[0001] The present disclosure relates generally to image processing, and more particularly, to a method and apparatus for performing depth estimation of objects in a scene. Background Art

[0002] Various electronic devices such as cameras, mobile phones and other multimedia devices are used to capture images of a scene. The depth map of the captured scene can be used in different applications such as robotics, automotive sensing, medical imaging and three-dimensional (3D) applications. A depth map is an image that includes information about the distance from a viewpoint to surfaces included in the scene.

[0003] Conventional camera systems have many processing bottlenecks in depth estimation and active depth sensors (e.g., depth estimation in oversaturated areas, depth estimation under varying lighting conditions, and depth estimation of reflective and transparent objects).

[0004] For example, in conventional complementary metal oxide semiconductor (CMOS) sensors and stereo setups used for depth estimation, accurate depth cannot be estimated in saturated image regions. In addition, conventional CMOS sensors cannot capture images of scenes at high frame rates and low power, which is not suitable for providing fast visual feedback.

[0005] In advanced driver assistance systems (ADAS), an accurate depth map of the scene is necessary for obstacle detection. In addition, the ADAS system should be able to operate under various lighting conditions and provide fast visual feedback to the user for correct navigation. However, conventional CMOS sensors do not operate well under various lighting conditions and cannot provide fast visual feedback to the user, which results in poor imaging characteristics of the depth map. In addition, conventional CMOS sensors require higher bandwidth. For example, conventional CMOS sensors can be sampled at the Nyquist rate, which requires more than 20GBPS.

[0006] Therefore, there is a need for a method and apparatus for performing accurate depth estimation of objects in a scene under various lighting conditions. Summary of the invention

[0007] Technical Solutions

[0008] According to one aspect of the present disclosure, a method for performing depth estimation of an object in an image by an electronic device is provided. The method includes: capturing an image of a scene by an electronic device; obtaining pixel intensity data and event data from the image by a sensor of the electronic device; generating an event depth map using the event data, wherein the event data includes event map data of the image and event speed data of the image; and generating a depth map of the object in the image using the event depth map and the pixel intensity data.

[0009] Beneficial Effects

[0010] The present disclosure provides a method and apparatus for performing accurate depth estimation of objects in an image of a scene under various lighting conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0012] Figure 1A An electronic device for performing depth estimation of an object in a scene according to an embodiment is shown;

[0013] Figure 1B shows a sparse depth map and a dense depth map according to an embodiment;

[0014] Figure 1C is a flow chart illustrating a method of obtaining an event intensity image according to an embodiment;

[0015] Figure 1D A method for obtaining scene data according to an embodiment is shown;

[0016] Figure 2 A depth estimation engine of an electronic device for performing depth estimation of an object in a scene according to an embodiment is shown;

[0017] Figure 3A is a flow chart illustrating a device method for performing depth estimation of an object in a scene according to an embodiment;

[0018] Figure 3B is a flow chart illustrating a method for generating a sparse event depth map according to an embodiment;

[0019] Figure 3C is a flow chart illustrating a method for creating a dense depth map according to an embodiment;

[0020] Figure 4is a flowchart showing a process of performing depth estimation of an object in a scene by an electronic device according to an embodiment;

[0021] Figure 5 is a flowchart showing a process of performing depth estimation of an object in a scene by an electronic device according to an embodiment;

[0022] Figure 6 is a flowchart showing a process of performing depth estimation of an object in a scene by an electronic device according to an embodiment;

[0023] Figure 7 is a flow chart illustrating a method for performing depth estimation of an object in a scene according to an embodiment;

[0024] Fig. 8A is a flowchart showing a process of performing timed synchronization of event data by an electronic device according to an embodiment;

[0025] Figure 8B illustrates timing synchronization of event data according to an embodiment;

[0026] Fig. 9 is a flow chart showing a process of performing frequency scaling by an electronic device according to an embodiment;

[0027] Fig.10 is a flowchart illustrating a process in which an electronic device performs obstacle detection and navigation of an autonomous driving vehicle by using a depth map according to an embodiment;

[0028] Fig.11 is a flowchart showing a process in which an electronic device creates a depth map to be used in augmented reality (AR) shopping according to an embodiment;

[0029] Fig.12 is a process flow diagram for reconstructing a 3D scene of a three-dimensional (3D) image according to an embodiment; and

[0030] Fig.13 is a flow chart of a process for creating a bokeh effect using a depth map according to an embodiment. DETAILED DESCRIPTION

[0031] The present disclosure is provided to address at least the above problems and / or disadvantages and to provide at least the advantages described below.

[0032] According to one aspect of the present disclosure, a method for performing depth estimation of an object in an image by an electronic device is provided. The method includes: capturing an image of a scene by an electronic device; obtaining pixel intensity data and event data from the image by a sensor of the electronic device; generating an event depth map using the event data, wherein the event data includes event map data of the image and event speed data of the image; and generating a depth map of the object in the image using the event depth map and the pixel intensity data.

[0033] According to one aspect of the present disclosure, there is provided an apparatus for performing depth estimation of an object in an image. The apparatus comprises: a camera configured to capture an image of a scene; a sensor configured to obtain pixel intensity data and event data from the image; and a processor configured to generate an event depth map using the event data, wherein the event data comprises event map data of the image and event velocity data of the image, and to generate a depth map of the object using the event depth map and the pixel intensity data.

[0034] According to one aspect of the present disclosure, a non-transitory computer-readable medium for storing instructions thereon is provided, wherein the instructions, when executed, instruct at least one processor to perform a method. The method includes: capturing an image of a scene by an electronic device; obtaining pixel intensity data and event data from the image by a sensor of the electronic device; using the event data to generate an event depth map, wherein the event data includes event map data of the image and event speed data of the image; and using the event depth map and the pixel intensity data to generate a depth map of an object in the image.

[0035] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. It includes various specific details to assist in that understanding, although these specific details are to be regarded as merely exemplary. Therefore, it will be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known functions and configurations have been omitted for clarity and brevity.

[0036] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are used by the inventor to convey a clear and consistent understanding of the present disclosure. Therefore, those skilled in the art will appreciate that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and not for the purpose of limiting the present disclosure as defined by the appended claims and their equivalents.

[0037] Herein, singular forms such as "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of such surfaces.

[0038] The various embodiments described herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0039] In this document, unless otherwise indicated, the term "or" refers to a non-exclusive or.

[0040] The examples used in this article are merely intended to promote understanding of the ways in which the embodiments herein can be practiced, and to further enable those skilled in the art to practice the embodiments herein. Therefore, the examples should not be interpreted as limiting the scope of the embodiments herein.

[0041] As is customary in the field of the present disclosure, embodiments may be described and illustrated in terms of blocks that perform one or more functions described. These blocks, which may be referred to herein as managers, engines, controllers, units, modules, etc., are physically implemented by analog and / or digital circuits (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc.), and may optionally be driven by firmware and software. The circuit may be embodied in one or more semiconductor chips, or on a substrate support such as a printed circuit board. The circuits constituting the blocks may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuits), or by a combination of dedicated hardware to perform some functions of the blocks and a processor to perform other functions of the blocks.

[0042] Without departing from the scope of the present disclosure, each block of the embodiment can be physically separated into two or more interacting and discrete blocks. Likewise, without departing from the scope of the present disclosure, the blocks of the embodiment can be physically combined into more complex blocks.

[0043] According to one embodiment, a method and apparatus for performing depth estimation of an object in a scene are provided.

[0044] A sparse event depth map can be generated by processing the event data of the scene.

[0045] A dense depth map of objects in a scene can be created by combining the sparse event depth map with a stream of pixel intensity data for the scene.

[0046] A spatiotemporal two-dimensional (2D) event map and an event intensity image may be generated by processing event map data of an image of a scene.

[0047] The sparse event depth map may be generated by processing the spatiotemporal 2D event map, event velocity data, and / or rotation of a capture device of the electronic device and translation estimation data of the sensor. The sensor may be a monocular event-based sensor of the electronic device.

[0048] The pixel intensity data stream of the scene can be processed along with the event data to obtain a high quality red, green, and blue (RGB) image of the scene.

[0049] An intermediate depth map may be generated by using the generated sparse depth and an event intensity image of the scene.

[0050] A dense depth map can be created by combining the intermediate depth map with a high-quality RGB image of the scene.

[0051] Timing synchronization and timing correction of event data are performed on the scene's pixel intensity data stream.

[0052] According to an embodiment, a method for performing depth estimation of an object in a scene by using an electronic device is provided. The method includes: capturing an image of a scene by an electronic device; and obtaining scene data, i.e., input data of the scene, by the electronic device from a sensor, wherein the input data includes a pixel intensity data stream of the scene and event data of the scene. The sensor may be a monocular event-based sensor. The method also includes: generating a sparse event depth map by the electronic device by processing the event data of the scene, wherein the event data of the scene includes event map data and event speed data; and generating a dense depth map of the object in the scene by the electronic device by fusing the sparse event depth map with the pixel intensity data stream of the scene.

[0053] Unlike conventional methods and systems, the method according to the embodiment can be used to perform depth estimation under various lighting conditions and in relation to reflective surfaces. For example, the method of the present disclosure can also be used to perform a more accurate 3D reconstruction of a scene by combining CMOS image sensor data with event sensor data using a single sensor (such as a monocular event-based sensor).

[0054] Due to the superior dynamic range of monocular event-based sensors, the methods of the present disclosure can be used to capture more information about a scene, thereby providing more information in saturated areas of the scene.

[0055] Unlike conventional methods and systems, the method of the present disclosure can be used to perform accurate depth estimation using the timing synchronization of CMOS image sensor data and event data. For example, low-power wearable visual devices (such as augmented reality (AR) smart glasses, human-computer interaction (HRI) devices, etc.) can use the timing synchronization of CMOS image sensor data and event data to perform accurate depth estimation of a scene.

[0056] Unlike conventional methods and systems, the method of the present disclosure can be used to reduce the dynamic random access memory (DRAM) bandwidth of a monocular event-based sensor by using a dynamic time event context rate based on frequency scaling. Therefore, low-latency processing based on reduced bandwidth can be obtained.

[0057] The method of the present disclosure can be used to generate a depth map with high accuracy using a single sensor such as a monocular event-based sensor. In addition, the method can be used to generate an accurate depth map without compromising the power consumption and performance of the electronic device.

[0058] Figure 1A An electronic device for performing depth estimation of an object in a scene according to an embodiment is shown.

[0059] refer to Figure 1A , the electronic device 100 may be, but is not limited to, a smart phone, a drone, a mobile robot, an autonomous vehicle, a smart watch, a laptop, a mobile phone, a head mounted display, a personal digital assistant (PDA), a tablet, a tablet phone, or any other electronic device including an image capture device. The electronic device 100 includes a monocular event-based sensor 110, a depth estimator 120, an image signal processor (ISP) 130, an event signal processor (ESP) 140, a communicator 150, a processor 160, a memory 170, and a display 180. The depth estimator 120, the ISP 130, the ESP 140, the communicator 150, and the processor 160 may be implemented as at least one hardware processor.

[0060] The monocular event-based sensor 110 may be configured to capture an image of a scene. The scene may be, but is not limited to, a 2D scene or a 3D scene.

[0061] The monocular event-based sensor 110 may be, but is not limited to, a camera, an RGB camera, a charge coupled device (CCD), or a CMOS sensor, etc.

[0062] The monocular event-based sensor 110 includes an active pixel sensor (APS) 111 and a dynamic vision sensor (DVS) 112. The APS 111 can be configured to obtain a pixel intensity data stream of a scene. The DVS 112 can be configured to obtain event data of an image. The monocular event-based sensor 110 can be configured to track motion changes in an image of a scene, i.e., changes in the intensity of pixels in the image of the scene.

[0063] The monocular event-based sensor 110 may be configured to perform the function of a depth sensor capable of obtaining depth information of an object in a 3D image of a scene, and the function of a motion sensor capable of acquiring motion information by detecting motion of an object in the 3D image of the scene.

[0064] The monocular event-based sensor 110 may obtain scene data while capturing an image of the scene using a capture device such as a camera of the electronic device 100. The scene data may include a pixel intensity data stream of the scene and event data of the scene.

[0065] Figure 1B A sparse event depth map and a dense depth map are shown according to an embodiment.

[0066] Throughout the specification, the term "sparse event depth map" may be used interchangeably with event depth map, sparse dense depth map, or sparse depth map. Likewise, the term "dense depth map" may be used interchangeably with depth map or dense map throughout the specification.

[0067] refer to Figure 1B , the event signal processing engine 140 can generate a sparse event depth map by processing the event data of the image. Generally, depth estimation involves two kinds of depth. One is sparse depth, which is the depth that is only known as a subset of the 2D image. The sparse event depth map 1003 can be depicted with multiple points obtained from the image 1001 of the scene. On the other hand, a dense depth map 1007 can be depicted with various depths known for the entire 2D image. The same color of the dense depth map indicates the same or substantially the same depth as the image 1001 of the scene as a 2D image.

[0068] The sparse event depth map 1003 and the pixel intensities may be used to obtain an intermediate dense depth map 1005. Throughout the specification, the term "intermediate dense depth map" may be used interchangeably with an intermediate map, a semi-dense depth map, or an intermediate depth map.

[0069] The event data of the image may include event graph data and event speed data of the image of the scene. The event graph data is data generated by accumulating event data in an address event representation (AER) format over a period of time and thereby generating a spatiotemporal 2D event graph of the event data. The event speed data refers to the predicted speed of each event using optical flow.

[0070] The event signal processing engine 140 may use the event map data of the image to generate a spatiotemporal 2D event map and an event intensity image.

[0071] Figure 1C is a flow chart illustrating a method of obtaining an event intensity image according to an embodiment.

[0072] refer to Figure 1C In step S101, event data is accumulated within a certain time period (eg, 30 ms).

[0073] In step S103, an event denoising process is applied to the accumulated event data.

[0074] In step S105, a surface of active events (SAE) is generated. SAE is a 3D domain including a two-dimensional sensor frame and an additional dimension representing time.

[0075] In step S107, intensity image smoothing is used to refine the initial reconstructed event intensity image by applying a smoothing function to the SAE.

[0076] In step S109, the final event intensity image is obtained after applying the smoothing function.

[0077] The event signal processing engine 140 can generate a sparse event depth map using the spatiotemporal 2D event graph, event velocity data, and camera rotation and translation estimation data of the monocular event-based sensor 110. For example, the sparse event depth map can be distributed along the edge of the image. The depth can indicate the relative distance between an object included in the scene and a capture device such as a camera.

[0078] When the event data is processed, the event signal processing engine 140 can process the pixel intensity data stream of the image of the scene and thereby obtain a high-quality RGB image from the captured image of the scene. The event signal processing engine 140 can use the generated sparse depth and pixel intensity data to generate an intermediate depth map. Typically, the pixel intensity data can be obtained by interpolating the accumulated event data.

[0079] Using the estimated pixel intensities, the sparse event depth map can be propagated to neighboring areas. That is, areas without depth information can be filled with the estimated pixel intensities to generate an intermediate depth map. The intermediate depth map can also be called a semi-dense (event) depth map.

[0080] The depth estimator 120 may generate a dense depth map by combining the intermediate depth map with the high-quality RGB image. The dense depth map may be generated by post-processing the intermediate depth map.

[0081] The depth estimator 120 can generate a dense depth map for objects in the image by combining the intermediate depth map with a high-quality RGB image of the scene using a guided surface fitting process. Since color, texture, and structural information about objects in the scene is incomplete based on the estimated event intensity, object information from the high-quality RGB image of the scene can be used to refine the intermediate event depth map.

[0082] ISP 130 may be configured to perform image signal processing using a guided surface fitting technique for creating a dense depth map.

[0083] ISP 130 may perform timing synchronization and timing correction of event data for a scene's pixel intensity data stream.

[0084] ESP 140 can be configured to perform frequency scaling on the depth estimation circuit of the electronic device 100 to determine an operating frequency for maintaining a balance between power and performance of the electronic device 100, wherein the frequency scaling is performed based on the scene data rate and the maximum throughput of the electronic device. The maximum throughput indicates the maximum amount of information that the electronic device 100 can process during a given amount of time. The operating frequency refers to the clock frequency at which a digital hardware block or digital hardware circuit operates. Frequency scaling is used to control the clock frequency to match the required performance and the power consumed.

[0085] The communicator 150 may be configured to communicate internally with hardware components in the electronic device 100. The processor 160 may be coupled to the memory 170 for processing various instructions stored in the memory 170 to perform depth estimation of an object in an image of a scene using the electronic device 100.

[0086] The memory 170 may be configured to store instructions to be executed by the processor 160. In addition, the memory 170 may be configured to store image frames of a 3D scene. The memory 170 may include a non-volatile storage element, such as a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or an electrically programmable memory (EPROM) or an electrically erasable programmable (EEPROM) memory.

[0087] In addition, the memory 170 may be a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. However, the term "non-transitory" should not be interpreted in the sense that the memory 170 is not removable. A non-transitory storage medium may store data that may change over time (e.g., random access memory (RAM) or cache).

[0088] Memory 170 may be configured to store larger amounts of information.

[0089] The display 180 may be configured to display an image of a captured 3D scene. The display 180 may include a touch screen display, an AR display, a virtual reality (VR) display, and the like.

[0090] Despite Figure 1A 100, but the embodiments of the present disclosure are not limited thereto. For example, the electronic device 100 may include: Figure 1A Two or more components may be combined together to perform the same or substantially similar functions of performing depth estimation of objects in a scene.

[0091] In addition, Figure 1A The labels or names of the components shown in the drawings are for illustration purposes only and do not limit the scope of the present disclosure.

[0092] Figure 1D A method for obtaining scene data according to an embodiment is shown.

[0093] refer to Figure 1D , when the camera captures a scene, in step S151, a plurality of photons are reflected from the scene, and in step S153, the reflected plurality of photons fall on a plurality of pixel arrays.

[0094] In step S155, each pixel may release several electron-hole pairs according to the wavelength of the photon. In step S157, the charges formed by the electron-hole pairs may be converted into a voltage using an electrical component such as a capacitor.

[0095] In step S159 , the voltage is compared with a previously stored voltage using the first comparator 191 .

[0096] In step S161 , the output of the comparator 191 , which indicates a voltage difference that can be converted into an intensity difference, is compared with a predetermined threshold value by the second comparator 193 .

[0097] In step S163 , the output of the second comparator 193 may be an on / off signal, which may be event data.

[0098] In step S165, a timestamp is attached to the event data, and the event data may be converted to AER data in an AER format. The event data may represent changes in pixel intensity in an image of a scene. That is, an image representing a scene may be divided into a plurality of units, which are referred to as "pixels". Each of a plurality of pixels in an image may represent a discrete region and have an associated intensity value. A pixel may appear dark at low grayscale intensity and may appear bright at high intensity.

[0099] In step S167, the voltage output associated with step S157 may be amplified using amplifier 195. The amplified voltage may be input into analog-to-digital converter (ADC) 197 in step S169 to generate a digital value representing the pixel intensity of the scene.

[0100] The event data may have the format of a time series of events e(n) from the monocular event-based sensor 110 as shown in equation (1) below.

[0101] e(n)={x n ,y n ,θ n ,t n}(1)

[0102] In equation (1), x and y refer to the coordinates of the pixel, θ refers to the polarity of the event, i.e., positive or negative, and t refers to the timestamp of the event triggering. A positive θ indicates that at the corresponding pixel, the intensity has increased by a certain threshold value Δ+>0 in the logarithmic intensity space. A negative θ indicates that the intensity has decreased by a second threshold value Δ->0 in the logarithmic intensity space.

[0103] Figure 2 A depth estimator of an electronic device for performing depth estimation of an object in an image of a scene according to an embodiment is shown.

[0104] refer to Figure 2 , the depth estimator 120 includes an event stream guided intensity generator 121, a sparse depth map generator 122, an intermediate depth map generator 123, an image guided surface fitting engine 124, a post-processor 125, and a dense depth map generator 126. The depth estimator 120 may be implemented as at least one hardware processor.

[0105] The event stream guided intensity generator 121 can generate intensity data from the event data of the monocular event based sensor 110. The sparse depth map generator 122 can generate a sparse event depth map using the event data of the image of the scene. The sparse depth map generator 122 can generate a spatiotemporal 2D event map and an event intensity image by processing the event map data of the image of the scene. In addition, the sparse depth map generator 122 can generate a sparse event depth map by processing the spatiotemporal 2D event map, event velocity data, and camera rotation and translation estimation data of the monocular event based sensor 110. The sparse event depth map can be distributed along the edges of the image of the scene.

[0106] When the event data is processed, the intermediate depth map generator 123 can simultaneously process the pixel intensity data stream of the image to obtain a high-quality RGB image of the scene. The intermediate depth map generator 123 can generate an intermediate depth map by using the generated sparse depth and the event intensity image of the scene.

[0107] The image-guided surface fitting engine 124 may create a dense depth map of objects in the image by fusing or combining the intermediate depth map with a high-quality RGB image from the scene using a guided surface fitting process.

[0108] The post-processor 125 may perform post-processing / depth smoothing for post-processing the intermediate depth map to generate a dense depth map.

[0109] The dense depth map generator 126 may create a dense depth map by fusing or combining the intermediate depth map with the high-quality RGB image. The dense depth map may be created by post-processing the intermediate depth map.

[0110] The dense depth map generator 126 may generate a dense depth map of objects in the image by fusing or combining the intermediate depth map with the high-quality RGB image using a guided surface fitting process. The guided surface fitting process may be used to find a best-fit line or surface for a series of data points using the high-quality RGB image and event data.

[0111] Figure 3A is a flow chart illustrating a device method for performing depth estimation of an object in a scene according to an embodiment.

[0112] refer to Figure 3A , the operations therein (ie, steps 310 - 340 ) may be performed by a processor or specifically by the depth estimator 120 of the electronic device 100 .

[0113] In step 310, the electronic device 100 captures an image of a scene. The capturing may be performed by a camera or a capture device of the electronic device 100.

[0114] In step 320 , the electronic device obtains scene data of the image through a sensor (eg, the monocular event-based sensor 110 of the electronic device 100 ).

[0115] In step 330, the electronic device 100 generates a sparse event depth map by using the event data of the image. The event data may include event map data and event speed data of the image of the scene.

[0116] To generate a sparse event depth map, other information such as camera orientation may be used along with the event data of the captured image. The relative position of the camera or capture device when capturing images of the scene and the flow of events between the accumulated image frames captured by the camera or capture device may help estimate the depth of the scene. Since events are available at certain pixels, specifically along the edges of the image of the scene, the generated event map is sparse.

[0117] In step 340, the electronic device 100 generates a dense depth map of the object in the image using the sparse event depth map and the pixel intensity data stream. The dense depth map may be generated by fusing or combining the sparse event depth map with the pixel intensity data stream of the image of the scene.

[0118] although Figure 3A The various steps, actions, behaviors, blocks, etc. in the flowcharts are shown in a particular order, but can be executed in a different order or simultaneously. In addition, some of the steps, actions, behaviors, blocks, etc. may be omitted, added, modified, or skipped without departing from the scope of the present disclosure.

[0119] Figure 3B is a flow chart illustrating a method for generating a sparse event depth map according to an embodiment.

[0120] refer to Figure 3B In step 321, the electronic device 100 generates a spatiotemporal 2D event map and / or an event intensity image by processing the event map data of the image of the scene. The sparse depth map generator 122 may generate a spatiotemporal 2D event map and / or an event intensity image using the event map data of the image.

[0121] In step 323, the electronic device 100 can generate a sparse event depth map by processing the spatiotemporal 2D event map and the event speed data. The spatiotemporal 2D event map and the event speed data can be used to generate a sparse event depth map. The rotation of the capture device (e.g., camera) of the electronic device 100 and the translation estimation data of the sensor 110 based on the monocular event can also be used to generate a sparse event depth map. The sparse event depth map can be distributed along the edge of the image. The sparse depth map generation engine 122 can use the aforementioned processing to generate a sparse event depth map.

[0122] although Figure 3B The various steps, actions, behaviors, blocks, etc. in the flowcharts are shown in a particular order, but they can be executed in different orders or simultaneously. In addition, some of the steps, actions, behaviors, blocks, etc. may be omitted, added, modified, or skipped without departing from the scope of the present disclosure.

[0123] Figure 3C is a flow chart illustrating a method for creating a dense depth map according to an embodiment.

[0124] refer to Figure 3C In step 331, the electronic device 100 obtains a high-quality RGB image from the image of the scene by simultaneously processing the pixel intensity data stream of the image while the event data is being processed. When the event data is being processed, the ISP 130 simultaneously processes the pixel intensity data stream of the image to obtain a high-quality RGB image from the image of the scene.

[0125] In step 333, the electronic device 100 generates an intermediate depth map based on the sparse event depth map and the event intensity image. The intermediate depth map generator 123 may generate an intermediate depth map using the generated sparse event depth map and the event intensity image of the scene.

[0126] In step 335, the electronic device 100 generates a dense depth map by combining or fusing the intermediate depth map with the high-quality RGB image of the scene. The dense depth map generator 143 may generate a dense depth map by combining or fusing the intermediate depth map with the high-quality RGB image.

[0127] although Figure 3C The various steps, actions, behaviors, blocks, etc. in the flowcharts are shown in a particular order, but can be executed in a different order or simultaneously. In addition, some of the steps, actions, behaviors, blocks, etc. may be omitted, added, modified, or skipped without departing from the scope of the present disclosure.

[0128] Figure 4 is a flow chart showing a process in which an electronic device performs depth estimation of an object in a scene according to an embodiment.

[0129] refer to Figure 4 The APS 111 may be, but is not limited to, a CMOS image sensor that can receive light and convert the received light into a digital pixel value depending on the light intensity. The sensor control circuit 110A may be, but is not limited to, a controller circuit that can be used to modify sensor parameter settings (such as exposure time, frame rate, etc.).

[0130] System phase locked loop (PLL) 135 may be a control system that generates an output signal whose phase is related to the phase of an input signal. System PLL 135 may be an application processor silicon on chip (AP SoC).

[0131] A CSI receiver (CSI Rx) 136 on the AP SoC may receive the APS data via the CSI D-Phy. In addition, the ISP 130 may be used to enhance the quality of the APS data.

[0132] DVS and APS feature correlation engine 131 may be configured to perform feature correlation between DVS data and APS data.Correction event timing engine 132 may be configured to perform timing synchronization and timing correction of event data for a pixel intensity data stream of a scene.

[0133] The event rate based frequency scaling engine 134 may be configured to perform frequency scaling on a depth estimation circuit of the electronic device 100 and determine an operating frequency of the electronic device 100 for maintaining a balance between power and performance, wherein the frequency scaling is performed based on an input data rate and a maximum throughput of the electronic device.

[0134] The event sensor timing synchronization engine 133 may be configured to perform timing synchronization of event data.

[0135] ESP 140 may include an AER bus decoder 141, an event camera application layer 142, an event stabilizer 143, an event temporal context detector 144, an event signal controller 149, an event buffer controller 145, an event graph generator 146, an event-based feature detector 147, and an event speed estimator 148.

[0136] The AER decoder 141 may receive event data from the DVS and change the event data from one clock domain (AER decoder 141) to another domain (event signal controller 149). The event camera application layer 142 may receive data from the AER decoder 141 and interface with the event signal controller 149. The event camera application layer 142 may also be used to filter out noise from the event data.

[0137] The event stabilizer 143 can fuse the attitude (or orientation) received from the sensor hub (e.g., an inertial measurement unit (IMU) or other directional sensor) and stabilize the event data relative to the sensor motion. The event time context detector 144 can determine the validity of the event on the time slice and determine the average accumulation time of the event data.

[0138] The event signal controller 149 may control event timing of event data and a digital intellectual property (IP) frequency control 250 of the event signal processor 140. Herein, the digital IP may be referred to as an "IP".

[0139] The event buffer controller 145 may manage a queue of accumulated event frames and handle internal memory requests.

[0140] An event map generator 146 may create a spatial 2D event map from single pixel event data. An event velocity estimator 148 may measure the velocity of events via particle tracking or other techniques. An event-based feature detector 147 may extract binary features from the event map.

[0141] Depth estimator 120 may receive data from ISP 130, corrected event timing engine 132, and event velocity estimator 148. By combining the data received from ISP 130, corrected event timing engine 132, and event velocity estimator 148, depth estimator 120 may create a depth map of objects in an image of a scene.

[0142] Figure 5 is a flow chart illustrating a process in which an electronic device performs depth estimation of an object in an image of a scene according to an embodiment.

[0143] refer to Figure 5 , in step S510 , the event sensor timing synchronization engine 133 receives the following inputs and sends the following outputs.

[0144] Input: event rate information, DVS event map, most recent time slice RGB image, system PLL clock information, RGB image V-Sync information, previously corrected event timing (starting from frame 2, for the first frame, assume constant).

[0145] Output: Corrected event timing (relative to the event signal controller).

[0146] In step S520 , the depth estimator 120 receives the following inputs and sends the following outputs.

[0147] Input: event graph, estimated camera pose, event velocity

[0148] Output: Event intensity image

[0149] In step S530 , the depth estimator 120 receives the following inputs and sends the following outputs.

[0150] Input: Event intensity image generated using guided event intensity generation engine

[0151] Output: Depth data based on sparse 3D edges

[0152] In step S540 , the depth estimator 120 receives the following inputs and sends the following outputs.

[0153] Input: Sparse 3D edge-based depth data from step 530

[0154] Output: Intermediate dense depth map using event sensor data

[0155] In step S550 , the depth estimator 120 receives the following inputs and sends the following outputs.

[0156] Input: RGB image from ISP 130, intermediate dense depth map from step S540

[0157] Output: Dense depth map from combined RGB and event-based data

[0158] In step S560 , the depth estimator 120 receives the following inputs and sends the following outputs.

[0159] Input: Depth map from step S550

[0160] Output: Post-processed smoothed dense depth map

[0161] In step S570, the event speed based frequency scaling engine 134 performs frequency scaling on the depth estimation circuit of the electronic device 100 to determine the operating frequency of the electronic device 100 for maintaining a balance between power and performance. The frequency scaling is performed based on the input data rate and the maximum throughput of the electronic device 100.

[0162] Figure 6 is a flow chart showing a process in which an electronic device performs depth estimation of an object in a scene according to an embodiment.

[0163] refer to Figure 6 , the depth estimator 120 can perform depth estimation of an image of a scene for saturated areas of the scene or under various lighting conditions of the scene. During image capture, for each scene, the monocular event-based sensor 110 can output pixel intensity data and event data (which represents a positive change or a negative change in pixel intensity). The event data is processed to generate a sparse depth map. The sparse depth map is processed to form an intermediate depth map. The pixel intensity data is processed in parallel by the ISP 130 to obtain a high-quality RGB image.

[0164] Guided surface fitting is applied to the intermediate depth map and the RGB image to obtain a dense depth map that represents the depth of objects more accurately than the intermediate depth map. Thereafter, post-processing such as depth smoothing and hole filling is applied. The output will be a dense, accurate depth map with the highest possible video resolution of the electronic device 100.

[0165] Unlike conventional stereo or active depth estimation methods, the event-based depth estimation of the present disclosure depends on camera / object motion. In addition, the low data rate makes the depth estimation similar to real-time depth measurement, which is effective for visual applications of wearable devices. In conventional methods, a pair of event sensor cameras are used to capture the depth of the scene. Unlike conventional methods, this method can use a single monocular event-based sensor 110 to capture the depth information of the scene.

[0166] Furthermore, the monocular event-based sensor 110 does not require calibration or correction for depth measurement. The CMOS and DVS sensor outputs are synchronized by the event sensor timing synchronization engine 133 of the present disclosure. The event-based sensor depth estimation disclosed herein produces a sparse depth map, where information is primarily distributed along object edges. Therefore, surface reflectivity does not affect depth measurement accuracy as is the case with stereo or active depth estimation. Image data from the APS 111 is used to obtain a dense depth map of the scene.

[0167] Figure 7 is a flowchart illustrating that an electronic device performs depth estimation of an object in an image of a scene according to an embodiment.

[0168] refer to Figure 7 In step 701, the monocular event-based sensor 110 starts capturing an event-based camera sequence. In addition, the monocular event-based sensor 110 performs processing for the DVS sensor data processing path and the CMOS sensor data processing path.

[0169] In step 702, ESP 140 begins to perform camera event stabilization, i.e., image stabilization, using techniques that reduce blur associated with the motion of a camera or other image capture device during exposure. Basically, an inertial measurement device can be used to measure the specific force and / or angular rate of the body to remove the effects of unnecessary camera motion from the input frame and obtain a stable image frame. The inertial measurement device can remove bias and / or temperature errors from the captured image to generate inertial measurement (IM) data. The IM data can be fused with event data that can be obtained from unstable images.

[0170] In step 703, ESP 140 checks the validity of each event across the time slice.

[0171] In step 704, ESP 140 identifies the average cumulative time for event map generation.

[0172] In step 705, ESP 140 generates an event map based on the identified average accumulation times, and in step 706, ESP 140 generates an intensity image from the event map.

[0173] In step 707, the ESP 140 performs event-based binary feature extraction from the generated event graph. In step 708, the ESP 140 uses particle tracking to measure event velocities, and in step 709, inputs the output of the event-based binary feature extraction and the event velocities to the DVS and APS feature correlation engine 131.

[0174] In step 711, RGB image data from the CMOS image sensor is processed. Steps 712-714 may be performed in parallel by the depth estimator 120 for creating a depth map.

[0175] In step 709 , the DVS and APS feature correlation engine 131 may receive input from the execution of event-based binary feature extraction, event velocity, and CMOS image data.

[0176] In step 718 , the event sensor timing synchronization engine 133 performs timing synchronization and transmits the result of the timing synchronization to the correction event timing engine 132 .

[0177] In step 719 , the corrected event timing engine 132 performs deep processing on the corrected event data timing event sensor data and performs event camera stabilization of the corrected event data timing event sensor data.

[0178] Fig. 8A is a flow chart showing a process of performing timed synchronization of event data by an electronic device according to an embodiment.

[0179] Typically, the monocular event-based sensor 110 is asynchronous in nature. Image data is represented in an AER format, which contains pixel coordinates, a timestamp of event generation, and an event type.

[0180] In the case of sensors that include dynamic pixels and active pixels, there are two separate interfaces for sending CMOS data via CSI D-PHY and event data via the AER bus interface. There are many technologies that are evolving to utilize event data and CMOS data. However, since they are generated and sent via two separate interfaces, timing synchronization and correction of event data is required for CMOS data of sensors that include dynamic pixels and active pixels.

[0181] refer to Fig. 8A , the timing synchronization can be mainly performed by the correlation engine 1300, wherein the correlation engine 1300 includes a DVS and APS feature correlation engine 131, an event sensor timing synchronization engine 133 and a correction event timing engine 132.

[0182] The timing synchronization engine 144 performs timing synchronization and correction of event data received from the monocular event-based sensor 110 including dynamic pixels and active pixels.

[0183] For timing synchronization, the RGB data from ISP 130 is processed to extract its image features. ISP 130 may also generate V-sync to indicate the end of the RGB frame completion. V-sync may be sent to a frame timing generator 137 based on V-Sync.

[0184] In step S801, the event graph from the event graph generator 146 may be processed to estimate event speed using the event speed estimator 148 and to detect event features using the event-based feature detector 147. In addition, timing information may be obtained via the system PLL 135. The output of the ISP 130 via the V-sync based frame timing generator 137 and the event-based feature detector 147 may be combined and fused by the DVS and APS feature correlation engine 131 to generate DVS data.

[0185] Based on the aforementioned feature correlation, in step S803, it is determined whether the feature correlation exceeds a predetermined threshold. Based on this determination, a timing correction amount to be applied on the DVS data can be determined.

[0186] Figure 8B Timing synchronization of event data according to an embodiment is shown.

[0187] refer to Figure 8B The correlation engine 1300 may combine and / or fuse the RGB data 810 obtained from the active pixel frame and the event data 820 obtained from the event data frame to correct and synchronize the time difference between the active pixel frame and the event data frame.

[0188] Fig. 9 is a flow chart showing a process of performing frequency scaling by an electronic device according to an embodiment.

[0189] refer to Fig. 9 , during the design of digital IP for camera-based systems, synthesis and timing convergence of maximum frequency by considering input data rate and maximum throughput is used. Unlike traditional CMOS-based sensors where the input rate is deterministic, monocular event-based sensors 110 are asynchronous in nature. Therefore, the rate at which data arrives and the required IP throughput vary.

[0190] However, in actual operation, the depth estimation IP should be clocked according to a criterion for maintaining an optimal balance between power and performance of the electronic device. The system of the present disclosure includes an event-rate-based frequency scaling engine 134 for the depth estimation IP, which can consider the input data rate and the current throughput and decide the IP operating frequency that can optimally balance power and performance.

[0191] The event rate based frequency scaling engine 134 receives output from the event rate estimation engine 148 and the event buffer controller 145 and processes the throughput to estimate the IP frequency.

[0192] Fig.10 is a process flow diagram illustrating obstacle detection and navigation by an autonomous vehicle using a depth map according to an embodiment.

[0193] refer to Fig.10 , the operations performed by elements 1000-1010 can be used by autonomous vehicles for obstacle detection or pedestrian tracking, especially in low-light conditions.

[0194] More specifically, the onboard IMU 1001 operates in conjunction with the wheel odometer 1003 to establish a global camera reference frame and its trajectory with respect to time. The pseudo image stream created by the event-based camera sensor 1002 is used to extract event-based features (such as corners and line segments), which are then compared with the IMU stabilization and positioning data obtained by the external localizer 1006. Thereafter, the accurate camera pose is estimated by the camera pose estimator 1007.

[0195] The external sensor calibrator 1004, the external position mapping unit 1005 and the external locator 1006 are configured to determine the exact position of the obstacle from the real-time depth information about the obstacle. In addition, the navigator 1010 and the external obstacle detector 1009 determine the existence of the obstacle from the real-time depth information.

[0196] The monocular event-based depth estimator 1008 is configured to operate with low power consumption and provide accurate depth information even in low light environments. The pseudo frame created by the event-based camera sensor 1002 gives a sparse estimate of the depth map that is further refined, and the holes in the depth map are filled and smoothed in real time with the time-correlated APS (RGB) frames.

[0197] Including Fig.10In an advanced driver assistance system (ADAS) that is an element of an ADAS system, an accurate depth map of the scene's image is necessary for obstacle detection. In addition, the ADAS system should work under various lighting conditions and provide fast visual feedback to the user for correct navigation. The monocular event-based sensor 110 provides a higher frame rate, a higher dynamic range (~120dB) under extreme lighting conditions.

[0198] Fig.11 is a process flow diagram of using depth maps in AR shopping according to an embodiment.

[0199] refer to Fig.11 The method can be used to determine an accurate depth map of an image of a scene including transparent and reflective surfaces and multiple objects. The operations performed by the elements in flowchart 1100 can be used for AR shopping and allow a user to perform virtual object placement based on depth map estimation. Fig.11 The approach can also be used in AR shopping use cases to employ real-time depth obtained from the approach in virtual reality object placement / realization.

[0200] The refined accurate depth information from the monocular event-based depth estimator 1104 is used for plane estimation by the plane region detector 1105. The monocular event-based depth estimator 1104 estimates the depth of objects in the image of the scene by fusing the active pixels and the event-based data. Thus, a 3D surface can be defined on which to project virtual objects.

[0201] The plane region detector 1105 is used to detect plane regions in the scene, thereby identifying suitable reserved locations for virtual object placement. The overlay rendering and seamless mixer 1106 is used to overlay virtual objects in suitable regions identified by the plane region detector 1105 .

[0202] Virtual components 1103 extracted from event-based camera sensor 1101 are warped about the desired estimate calculated from depth and plane information. The data is fused and the corresponding virtual components are overlaid on the rendering surface and seamlessly blended 1106. The resulting image is rendered with other image data as seen on the user's viewfinder / preview, providing an AR experience via augmented image 1108 displayed on display 1109.

[0203] Fig.12 is a flow chart of a process for reconstructing a 3D scene of a 3D image according to an embodiment.

[0204] refer to Fig.12 , operations corresponding to elements 1201 - 1208 in flowchart 1200 may be used to reconstruct a 3D scene of a 3D image.

[0205] Depth maps are important for 3D imaging and display, and can be used in different application areas, such as digital holographic image processing, object reconstruction in holistic imaging, 3D object retrieval and scene understanding, and 3D printing. In addition, 3D depth maps are used in vision-based applications, such as in robotics, gaming, AR glasses, etc.

[0206] The accurate depth map of objects in the scene obtained by using the monocular event-based sensor 110 is used for virtual object placement. The accurate depth map can be used in product previews to visualize or understand different products and features.

[0207] Fig.13 is a flow chart of a process for creating a bokeh effect using a depth map using an electronic device according to an embodiment.

[0208] refer to Fig.13 , steps 1301 and 1303 may be used to create a bokeh effect using a depth map. The electronic device 100 may capture an image of a scene, and based on processing via a bokeh engine 1301 and a depth estimator 1302 using a monocular event-based sensor, an accurate depth map of the image of the scene may be created.

[0209] This process can be used to create a bokeh effect. Therefore, the real-time depth map obtained according to the present disclosure can be used to apply a high-quality bokeh effect to a still image. A device (e.g., a smart camera platform) can capture a series of images presented to a user's preview / viewfinder. The event data from the event pipeline is processed to obtain a fine-grained, accurate and real-time depth map via a depth estimator 1302 using a monocular event-based sensor. The depth map can be used to obtain a solution (such as a bokeh effect) to decompose a scene into multiple depth regions and classify multiple regions into layers of background or foreground based on the region of interest. In addition, such a solution can be used to apply various image processing kernels to achieve the desired results.

[0210] The embodiments disclosed herein may be implemented by at least one software program running on at least one hardware device and performing network management functions to control the various elements.

[0211] In Figures 1 to Fig.13 The elements shown in the EMBODIMENTS include blocks, which may be at least one of a hardware device, a software module, or a combination of a hardware device and a software module.

[0212] While the present disclosure has been particularly shown and described with reference to certain embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. A method for performing depth estimation of an object in an image by an electronic device, the method comprising: capturing an image of the scene by an electronic device; Obtaining pixel intensity data and event data from the image by a sensor of the electronic device; Generating an event depth map using the event data, wherein the event data includes event map data of the image and event velocity data of the image, wherein generating the event depth map includes: generating a spatiotemporal two-dimensional 2D event map by processing the event map data of the image, and generating the event depth map using the spatiotemporal 2D event map, the event velocity data, a rotation of a capture device of the electronic device, and a translation estimation data of a sensor of the electronic device; and The event depth map and the pixel intensity data are used to generate a depth map of objects in the image. 2 . The method of claim 1 , further comprising distributing the event depth map along edges of the image.

3. The method according to claim 1, wherein: Generating a depth map for an object in an image using the event depth map and the pixel intensity data includes combining the event depth map with the pixel intensity data.

4. The method according to claim 1, wherein: Generating a depth map of an object in an image involves: Obtain red, green and blue RGB images from the image using pixel intensity data; generating an intermediate depth map based on the event depth map and the event intensity image; and A depth map is generated by combining the intermediate depth map with the RGB image.

5. The method according to claim 4, wherein: Combining the intermediate depth map with the RGB image includes combining the intermediate depth map with the RGB image by applying a guided surface fitting process.

6. The method according to claim 4, wherein: Generating a depth map of an object in the image also includes post-processing the intermediate depth map using depth smoothing and hole filling of the intermediate depth map.

7. The method according to claim 1, wherein: The pixel intensity data and event data obtained by the sensor of the electronic device include: obtaining pixel intensity data from an active pixel sensor (APS) of the sensor; and The event data is obtained by the dynamic vision sensor DVS of the sensor.

8. The method of claim 1 further comprising performing timing synchronization and timing correction of event data with respect to pixel intensity data.

9. The method according to claim 1, further comprising: performing frequency scaling on a depth estimation circuit of the electronic device based on a scene data rate of the data and a maximum throughput of the electronic device; as well as An operating frequency of the electronic device is determined based on the frequency scaling.

10. The method according to claim 1, further comprising: The sensor obtains depth information of the object in the image; as well as The motion information of the object in the image is obtained by the sensor.

11. An apparatus for performing depth estimation of an object in an image, the apparatus comprising: a camera configured to capture an image of a scene; a sensor configured to obtain pixel intensity data and event data from an image; as well as The processor is configured as: Generating an event depth map using event data, wherein the event data includes event map data of the image and event velocity data of the image, wherein generating the event depth map includes: generating a spatiotemporal two-dimensional 2D event map by processing the event map data of the image, and generating the event depth map using the spatiotemporal 2D event map, the event velocity data, a rotation of a capture device of the electronic device, and a translation estimation data of a sensor of the electronic device, and A depth map of the object is generated using the event depth map and the pixel intensity data.

12. A non-transitory computer-readable medium having instructions stored thereon, wherein the instructions, when executed, direct at least one processor to perform a method comprising: capturing an image of the scene by an electronic device; Obtaining pixel intensity data and event data from the image by a sensor of the electronic device; Generating an event depth map using the event data, wherein the event data includes event map data of the image and event velocity data of the image, wherein generating the event depth map includes: generating a spatiotemporal two-dimensional 2D event map by processing the event map data of the image, and generating the event depth map using the spatiotemporal 2D event map, the event velocity data, a rotation of a capture device of the electronic device, and a translation estimation data of a sensor of the electronic device; and The event depth map and the pixel intensity data are used to generate a depth map of objects in the image.

Images