System and method for laser depth map sampling

Abstract

The present invention discloses a method for automatic control of a vehicle, comprising: generating a first plurality of spatial coordinate sets based on content analysis of a first received image, each spatial coordinate set indicating a position of a fragment within a first object in the first received image, wherein at least a portion of the fragment has a boundary along a boundary of the first object; providing the first plurality of spatial coordinate sets to a LIDAR unit to selectively obtain information indicating depth at a position representing the first plurality of spatial coordinates outside the vehicle in a scene; determining a position of the first object relative to the vehicle based on the first plurality of spatial coordinate sets and depth information received from the LIDAR unit; and determining instructions for vehicle operation based on the determined position of the first object. The present invention also discloses a method for capturing images using an electronic device. The present invention also discloses several other aspects.

Description

[0001] This application is a divisional application of the invention patent application with application date of July 11, 2016, application number 201680046908.3 and name “SYSTEM AND METHOD FOR LASER DEPTH MAP SAMPLING”. Technical Field

[0002] The present invention relates generally to electronic devices. More specifically, the present invention relates to systems and methods for depth map sampling. Background Art

[0003] Over the past several decades, the use of electronic devices has become commonplace. In particular, advances in electronic technology have reduced the cost of increasingly complex and useful electronic devices. Cost reductions and consumer demand have proliferated the use of electronic devices such that they are virtually ubiquitous in modern society. As the use of electronic devices has expanded, so has the demand for new and improved features of electronic devices. More specifically, electronic devices that perform new functions and / or perform functions faster, more efficiently, or with higher quality are often sought after.

[0004] Some electronic devices (e.g., cameras, video camcorders, digital cameras, cellular phones, smart phones, computers, televisions, etc.) can create depth maps using LIDAR (light + radar) scanning. Densely sampling a scene using LIDAR scanning is expensive in terms of time and power. This can result in low frame rates and battery drain. As can be observed from this discussion, systems and methods that improve LIDAR depth map sampling can be beneficial. Summary of the invention

[0005] The present invention describes an electronic device. The electronic device includes a camera configured to capture an image of a scene. The electronic device also includes an image segmentation mapper, which is configured to perform segmentation on the image based on the image content to generate a plurality of image segments, each of the plurality of image segments is associated with a spatial coordinate indicating the position of each segment in the scene. The electronic device further includes a memory configured to store the image and the spatial coordinates. The electronic device additionally includes a LIDAR (Light + Radar) unit, which can be turned to selectively obtain depth values ​​corresponding to at least a subset of the spatial coordinates. The electronic device further includes a depth mapper configured to generate a depth map of the scene based on the depth values ​​and the spatial coordinates.

[0006] At least a portion of the image segments include non-uniform segments defining boundaries of objects within the image. The image segmentation mapper may be configured to perform segmentation based on image complexity. The number of generated segments may vary with the determined complexity of the image.

[0007] The LIDAR unit may be configured to perform a coarse scan of an area of ​​the scene containing substantially homogeneous objects within the scene.The image segmentation mapper may be configured to provide spatial coordinates to the LIDAR unit and may be configured to provide segments of the image to the depth mapper.

[0008] The depth mapper may be configured to generate a depth map by merging the segments with the corresponding depth values.The depth mapper may be configured to generate a depth map by filling each segment with the corresponding depth value obtained by the LIDAR unit at the spatial coordinates.

[0009] The plurality of depth values ​​obtained by the LIDAR unit may be configured to be adjusted based on feedback from previous depth maps.The spatial coordinates of a segment may correspond to the centroid of the segment.

[0010] The present invention also describes a method. The method includes capturing an image of a scene. The method also includes performing segmentation on the image based on image content, each of the plurality of segments being associated with a spatial coordinate indicating a location of each segment in the scene. The method further includes obtaining depth values ​​corresponding to at least a subset of the spatial coordinates by a LIDAR (Light + Radar) unit. The LIDAR unit can be steered to selectively obtain the depth values. The method additionally includes generating a depth map of the scene based on the depth values ​​and the spatial coordinates.

[0011] The present disclosure also describes an apparatus. The apparatus includes means for capturing an image of a scene. The apparatus also includes means for performing segmentation of the image based on image content, each of the plurality of image segments being associated with a spatial coordinate indicating a location of each segment in the scene. The apparatus further includes means for obtaining depth values ​​corresponding to at least a subset of the spatial coordinates by a LIDAR unit. The LIDAR unit can be steered to selectively obtain the depth values. The apparatus additionally includes means for generating a depth map of the scene based on the depth values ​​and the spatial coordinates.

[0012] The present invention also describes a computer program product. The computer program product includes a non-transitory tangible computer-readable medium having instructions thereon. The instructions include code for causing an electronic device to capture an image of a scene. The instructions also include code for causing the electronic device to perform segmentation of the image based on the image content to generate a plurality of image fragments, each of the plurality of image fragments being associated with a spatial coordinate indicating the location of each fragment in the scene. The instructions further include code for causing the electronic device to obtain depth values ​​corresponding to at least a subset of the spatial coordinates through a LIDAR unit. The LIDAR unit can be turned to selectively obtain the depth values. The instructions additionally include code for causing the electronic device to generate a depth map of the scene based on the depth values ​​and the spatial coordinates. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A block diagram illustrating an electronic device configured to perform image-assisted LIDAR (Light + Radar) depth map sampling;

[0014] Figure 2 is a block diagram illustrating another configuration of an electronic device configured to perform image-assisted LIDAR depth map sampling;

[0015] Figure 3 A flowchart illustrating a method for performing image-assisted LIDAR depth map sampling;

[0016] Figure 4 A flow chart illustrating another method for performing image-assisted LIDAR depth map sampling;

[0017] Figures 5A to 5C An example of generating a segmentation map from an image is shown;

[0018] Figures 6A to 6D An example of image-assisted LIDAR depth map sampling is described;

[0019] Figures 7A to 7D Another example of image-assisted LIDAR depth map sampling is described;

[0020] Figure 8 is a block diagram illustrating one example of an electronic device in which systems and methods for image-assisted LIDAR depth map sampling may be implemented; and

[0021] Fig. 9 Some of the components that may be included in an electronic device are described. DETAILED DESCRIPTION

[0022] In many applications, it is important to generate a depth map of objects in a scene. For example, with advanced driver assistance systems (ADAS), it is important to identify the drivable area in front of the vehicle to enable obstacle avoidance. Obstacle detection can use depth maps to identify traffic signs (e.g., speed limit signs, stop signs, road signs, etc.). Another scenario where depth maps are important is vehicle automation where an autonomous vehicle (e.g., an unmanned aerial vehicle (UAV) or an autonomous car) senses the environment and navigates without human input.

[0023] Depth map generation can also be used in various production and entertainment scenarios. For example, a depth map of a room or environment can be used for social, augmented or virtual reality, 3-D reconstruction, safety and security, and similar applications. Further, depth can be referenced for image and / or video capture to, for example, assist in autofocus operations. In these and other scenarios, it can be advantageous to intelligently acquire a depth map that accurately and precisely captures depth information of a scene, room, or environment.

[0024] LIDAR (Light + Radar) scanning can be used to sample the depth value of an object. In LIDAR, a laser can illuminate an object and the distance of the object from a reference position (i.e., the depth value) can be determined by analyzing the reflected light. Dense sampling of a scene is expensive in terms of time and power, resulting in low frame rates and battery drain. However, uniform (i.e., simple) LIDAR scanning at regular points can result in poor quality depth maps. Similarly, even when a uniform or predefined scanning pattern that does not take into account the content of the scene is adopted, dense sampling may still miss small objects, resulting in depth map errors.

[0025] In the systems and methods described herein, instead of performing a LIDAR scan based on uniform scan points, the LIDAR unit may perform image-assisted LIDAR depth map sampling. A camera positioned near the LIDAR unit may capture one or more images of a scene. This image or set of images may be segmented. The fragments may follow the boundaries of an object in the image, thereby preserving the edges of the object. For example, the spatial coordinates (X, Y position) of the centroid of the fragment may be provided to the LIDAR unit, which performs a LIDAR scan at those spatial coordinates. The depth values ​​generated by the LIDAR unit from the scan may be filled in the corresponding fragment to create a depth map of the scene. For example, each fragment may be given a uniform depth associated with the depth of the corresponding fragment centroid. The following explains in more detail the systems and methods for image-assisted LIDAR depth map sampling.

[0026] Figure 1 A block diagram is provided to illustrate an electronic device 102 configured to perform image-assisted LIDAR depth map sampling. The electronic device 102 may also be referred to as a wireless communication device, a mobile device, a mobile station, a subscriber station, a client, a client station, a user equipment (UE), a remote station, an access terminal, a mobile terminal, a terminal, a user terminal, a subscriber unit, etc. Examples of electronic devices include laptop or desktop computers, cellular phones, smart phones, wireless modems, e-readers, tablet devices, gaming systems, robots, aircraft, unmanned aerial vehicles (UAVs), automobiles, etc. Some of these devices may operate according to one or more industry standards.

[0027] In many scenarios, the electronic device 102 can use the depth map 120 of the scene. In one example, a smartphone can generate the depth map 120 of the scene. In another example, a car can include an advanced driver assistance system (ADAS) that can use the depth map 120 to adjust the speed, steering, parking, etc. of the car based on detected traffic signs, signals, and / or other objects. In another example, an unmanned aerial vehicle (UAV) can generate a depth map 120 from a video recorded while in flight, can navigate based on detected objects (e.g., buildings, signs, people, packages, etc.), can pick up and / or deliver detected packages, etc. Many other examples can be implemented according to the systems and methods disclosed herein. For example, the systems and methods disclosed herein can be implemented in a robot that performs one or more actions (e.g., pick up something, assemble something, find an item, etc.) based on one or more objects detected using the depth map 120.

[0028] For example, an electronic device 102 (e.g., a smart phone or tablet computer) may include a camera 104. The camera 104 may include an image sensor 108 and an optical system 106 (e.g., a lens) that focuses an image of an object within the field of view of the optical system 106 on the image sensor 108. The camera 104 may be configured to capture a digital image. In an embodiment, the digital image may be a color image. Although the systems and methods of the present invention are described in terms of captured images, the techniques discussed herein may be used for any digital image. Therefore, the terms video frame and digital image are used interchangeably herein. Likewise, in certain embodiments, the electronic device 102 may not include a camera 104 and an optical system 106, but may receive or utilize a stored digital image.

[0029] The electronic device 102 may also include a camera software application and a display screen. When the camera application is running, images of objects within the field of view of the optical system camera 104 may be recorded by the image sensor 108. The images recorded by the image sensor 108 may be displayed on the display screen. These images may be displayed continuously at a relatively high frame rate so that, at any given moment, objects within the field of view of the camera 104 are displayed on the display screen.

[0030] The user interface 112 of the camera application may allow selection of one or more objects displayed on the display screen. In one configuration, the display is a touch screen 110 that receives input from physical touch (e.g., by a finger, stylus, or other tool). The touch screen 110 may receive touch input that defines a target object. The target object may be selected in any suitable manner. For example, facial recognition, pedestrian recognition, etc. may be used to select the target object.

[0031] The electronic device 102 may also include a LIDAR unit 114. LIDAR (Light + Radar) is a remote sensing technology that measures distance by illuminating a target with laser light and analyzing the reflected light. The LIDAR unit 114 can sample a scene to determine depth values ​​of objects within the scene. Thus, the LIDAR unit 114 may include a light generating (laser) portion 137 and a light receiving (sensor) portion 131. The LIDAR unit may further include various optical components 141 to facilitate generating and receiving light and additional features as described below.

[0032] In an implementation, the LIDAR unit 114 may sense a laser pulse in a certain direction and may measure the time it takes for the laser pulse to return to the LIDAR unit 114. This process is similar to a radar that sends a signal and receives an echo. The LIDAR unit 114 may record the time difference between sending the laser and receiving the reflected light. Based on this time difference, the LIDAR unit 114 may obtain a depth value representing a distance estimate. In some implementations, the time measurement, recording, and / or obtaining the depth value may be performed external to the LIDAR unit 114, for example, in one or more processors 142 within the electronic device 102 or remote from the electronic device. In some implementations, the memory 144 may store information associated with image-assisted LIDAR depth map sampling (e.g., images, depth values, fragments, and depth maps 120).

[0033] The laser of the LIDAR unit 114 can be directed to different locations, such as locations identified by an image segmentation process. Thus, the laser can be steerable. In one approach, the laser can be mechanically directed. For example, the laser can be directed with a mirror or an actuator. In another example, the laser can be directed via an optical element that can be electronically steered. In this approach, the LIDAR unit 114 can have no moving parts and provide more flexibility.

[0034] The dense sampling of the scene by the LIDAR unit 114 is expensive in terms of time and power. The above situation may reduce the frame rate (e.g., the number of images captured by the camera 104) that the electronic device 102 can handle. In addition, the above situation is particularly important for platforms that use batteries. Sending light pulses consumes a lot of power. For mobile platforms (e.g., mobile phones) that are powered by batteries, sampling the surrounding scene at a very high rate can quickly drain the battery. The time-consuming acquisition and / or generation of depth maps may also be more important in platforms that are not power constrained or have fewer power constraints than mobile platforms. For example, in the case where the scene from which the depth information is captured is dynamic (e.g., various (manned and unmanned) land and air vehicles, robots, etc.), speed may be advantageous.

[0035] In addition, there is the problem of determining the best position to perform the LIDAR scan. Some approaches can perform uniform LIDAR scans. For example, the rotating mirror approach can obtain equal sampling of the scene, where the LIDAR scan is performed in equal or uniform increments. This approach can be called simple LIDAR scanning. In this approach, the mirror rotates at a certain speed and the LIDAR sends laser pulses every several milliseconds. The uniform speed of rotation and uniform laser pulses give uniform sampling of the scene, i.e., a uniform or other predefined two-dimensional grid of points across the horizontal and vertical range of the scene. However, this approach may not detect certain objects within the scene. For example, uniform LIDAR sampling may not detect thin objects between the sampling points.

[0036] The systems and methods described herein provide image-assisted LIDAR depth map sampling. The LIDAR unit 114 may perform LIDAR scans assisted by images captured by the camera 104. The LIDAR unit 114 may adapt depth sampling in response to scene content as captured by the images.

[0037] In an embodiment, the electronic device 102 may include an image segmentation mapper 116. The camera 104 may provide the captured image to the image segmentation mapper 116. The image segmentation mapper 116 may segment the image to generate a segmentation map. The segmentation map may include a plurality of segments. The centroids of the segments may have spatial coordinates. The image segmentation mapper 116 may send the spatial coordinates of the centroids of the segments to the LIDAR unit 114. The image segmentation mapper 116 may also provide the segments to the depth mapper 118.

[0038] The LIDAR unit 114 may perform a LIDAR scan based on the received spatial coordinates. In an embodiment, the LIDAR unit 114 may use a steerable LIDAR to sample the depth of the scene at the centroid of each segment. The LIDAR unit 114 and the camera 104 may be positioned near each other. The LIDAR unit 114 and the camera 104 may be aligned so that they have substantially the same field of view (FOV). The LIDAR unit 114 and the camera 104 may be calibrated relative to each other. For example, the zoom or the camera 104 and the LIDAR unit 114 may be calibrated.

[0039] For accurate scene coordinates, the system may use sampled coordinates (e.g., centroids), calibration parameters, and previous depth maps. For example, the LIDAR unit 114 may steer a laser to determine a depth value of the scene at the spatial coordinates of the centroid. In one embodiment, the LIDAR unit 114 may determine the depth value and send the depth value to the depth mapper 118. In another embodiment, the LIDAR unit 114 may send raw data (e.g., time measurements) to the processor 142 for determining the depth value. The processor 142 may then provide the depth value to the depth mapper 118.

[0040] Upon receiving the segments from the image segmentation mapper 116 and the depth values ​​from the LIDAR unit 114, the depth mapper may fill each segment with the depth value of the centroid of each segment to create a depth map 120. Figure 2 More details are given on image-assisted LIDAR depth map sampling.

[0041] The described systems and methods provide depth sampling that is adaptive to the content of the scene. For example, the LIDAR unit 114 may be instructed to perform a relatively coarse scan (fewer points per unit area) for areas of the scene that contain large and substantially uniform objects within an image of the scene, while the LIDAR unit 114 may be instructed to perform a relatively fine scan (more points per unit area) for areas of the scene that contain objects that are smaller, vary in geometry, and / or include multiple adjacent or proximate objects.

[0042] Thus, by adaptively generating a scan pattern based on scene content, a depth map can be efficiently obtained without sacrificing accuracy. In fact, by capturing points at objects or parts of objects that might be missed using uniform scanning, for example, accuracy (i.e., a depth map 120 of a scene that is close to ground truth) can be improved over uniformly spaced scans. This results in a higher accuracy depth map 120 and a higher frame rate. Because the LIDAR scan is optimized based on the captured image, this results in less power consumption, which is particularly important for battery-powered applications.

[0043] Figure 2 1 is a block diagram illustrating another configuration of an electronic device 202 configured to perform image-assisted LIDAR depth map sampling. The electronic device 102 may include a camera 204, an image segmentation mapper 216, a LIDAR unit 214, and a depth mapper 218. The camera 204, the image segmentation mapper 216, the LIDAR unit 214, and the depth mapper 218 may be a combination of the above. Figure 1 The configurations of the camera 104 , image segmentation mapper 116 , LIDAR unit 214 , and depth mapper 118 are described.

[0044] The camera 204 may be positioned on or near the LIDAR unit 214. For example, the camera 204 may be mounted on the electronic device 202 near the LIDAR unit 214. The camera 204 and the LIDAR unit 214 may be oriented so that they have approximately the same view of the scene 222. The scene 222 may include one or more objects at different locations. Thus, the one or more objects may be at the same or different depths (e.g., distances) from the camera 204 and the LIDAR unit 214.

[0045] Camera 204 may capture image 224 of scene 222. Image 224 may be a digital color image. Color images may provide better segmentation performance than grayscale images. In the example of an image of a red ball on green grass, the image may be easily segmented in the color domain, but may result in inaccurate segmentation boundaries because the grayscale of the grass and the ball are similar.

[0046] The image 224 may be composed of a plurality of pixels. Each pixel may have a spatial coordinate 232. In an implementation, the spatial coordinate 232 may be expressed as a Cartesian coordinate (eg, an xy coordinate), where x corresponds to the horizontal position of the pixel and y corresponds to the vertical position of the pixel.

[0047] The camera 204 can provide the image 224 to the image segmentation mapper 216. The image segmentation mapper 216 can generate a segmentation map 226 from the image 224. The segmentation map 226 can be composed of a plurality of segments 228. Each segment 228 can be identified by a spatial coordinate 232. For example, the segment 228 can be identified by the spatial coordinate 232 associated with its centroid 230.

[0048] The image segmentation mapper 216 may use low-level computer vision techniques to generate the segmentation map 226. The segments 228 may be determined based on the content of the image 224. The segments 228 may be non-uniform, having irregular geometry, size, and distribution in response to the content of the image 224. One or more segments 228 may define the boundaries of an object in the image 224. The boundaries of the segments 228 follow the boundaries or edges of the objects from the image 224. For example, the image segmentation mapper 216 may determine the location of the boundaries between objects in the image 224. The segments 228 may be generated to preserve the boundaries (e.g., edges) between objects in the image 224. An example of a segmentation map 226 is described in conjunction with FIG. 5.

[0049] The image segmentation mapper 216 may determine spatial coordinates 232 for each segment 228 identified in the segmentation map 226. The spatial coordinates 232 indicate the location of each segment 228 in the scene 222. In one embodiment, the image segmentation mapper 216 may determine the spatial coordinates 232 of the centroid 230 of each segment 228. The centroid 230 of a given segment 228 is the geometric center of the segment 228. The centroid 230 may be determined as the arithmetic mean (e.g., average) location of all points in the segment 228. The centroid 230 of a segment 228 has associated spatial coordinates 232. For example, the centroid 230 may be expressed in terms of an x ​​(e.g., horizontal) coordinate and a (e.g., vertical) coordinate.

[0050] The spatial coordinates 232 of the segment 228 may be identified by other approaches. In one approach, the spatial coordinates 232 of the segment 228 may correspond to the center of a bounding box around the segment 228. In another approach, if the segment 228 is calculated using a clustering algorithm (e.g., K-means), the cluster center at the last iteration may be used as a sampling point (e.g., spatial coordinates 232) for the segment 228. In another approach, in the case of morphological erosion of the segment 228, the last point position is the sampling position (e.g., spatial coordinates 232).

[0051] The number of segments 228 generated by the image segmentation mapper 216 may be determined according to different approaches. In one approach, the number of segments 228 may be a fixed number (e.g., 2,000 segments). In another approach, the number of segments 228 may be a maximum number. In another approach, the number or quantity of segments 228 may vary based on feedback 240 from the generated depth map 220, as described below. In another approach, the number or quantity of segments 228 may be determined to vary with image content, for example, a complex scene 222 (e.g., many objects, small objects, varying objects) may utilize a greater number of scenes 228 relative to less complex scenes (e.g., fewer objects, large objects, uniform objects). Various approaches (indicative of metrics of: number of edges, edge length, color, color difference, etc.) may be utilized to determine the complexity of the scene that may be used to derive the number of segments 228.

[0052] The image segmentation mapper 216 can provide the spatial coordinates 232 of the segments 228 to the LIDAR unit 214. For example, the image segmentation mapper 216 can send the spatial coordinate information 233 including the spatial coordinates 232 of the centroid 230 to the LIDAR unit 214. The image segmentation mapper 216 can also provide the segments 228 of the image 224 to the depth mapper 218. For example, the image segmentation mapper 216 can send the segment information 229 to the depth mapper 218.

[0053] LIDAR unit 214 may perform a LIDAR scan based on spatial coordinates 232 of segment 228. Therefore, spatial coordinates 232 may be sampling points at which LIDAR unit 214 performs the LIDAR scan.

[0054] The LIDAR unit 214 may include a steerable laser 234 and a depth value determination block 236. The steerable laser 234 may be calibrated to move according to the spatial coordinates 232 of the image 224. Thus, the direction of the steerable laser 234 may be coordinated with the spatial coordinates 232 of the image 224. The camera 204 and the steerable laser 234 may be calibrated so that the spatial coordinates 232 of the image may be converted to the position of the steerable laser 234. Thus, by aligning and calibrating the camera 204 and the steerable laser 234, the electronic device 202 may determine the conversion from image space to LIDAR space.

[0055] In some implementations, the LIDAR unit 214 may receive scanning instructions from a processor, such as the processor 242. The scanning instructions may include commands and / or information regarding positioning, steering, orienting, or otherwise enabling movement of the LIDAR unit 214 to sample the scene 222. In other implementations, the LIDAR unit 214 may receive the segments 228 or the images 224 and generate the segments 228 and / or the centroids 230.

[0056] It should be noted that depth values ​​238 may be transformed from LIDAR space to image space. Depth data (e.g., depth values ​​238) are coordinates in a world coordinate system (i.e., XYZ space). These 3D coordinates may be projected onto image 224 (i.e., image space) via a projective transformation. The transformation may be estimated from, for example, a calibration target.

[0057] Steerable laser 234 may be steered to take samples of scene 222 at points corresponding to spatial coordinates 232 of segment 228. It should be noted that because the LIDAR scan is based on spatial coordinates 232 of segmentation map 226, the LIDAR scan may be a non-uniform scan.

[0058] The depth value determination block 236 may determine a depth value 238 at the spatial coordinate 232 of the segment 228. This may be done by analyzing the amount of time it takes for light to reflect from an object in the scene 222 and be received. The LIDAR unit 214 may provide the depth value 238 to the depth mapper 218. In another embodiment, the LIDAR unit 214 may provide data (e.g., temporal data indicating the depth value) to the processor 242 to calculate the depth value 238.

[0059] The depth mapper 218 may create the depth map 220 by merging the segments 228 provided by the image segmentation mapper 216 with the depth values ​​238 provided by the LIDAR unit 214. In one approach, the depth mapper 218 may populate each segment 228 with the depth value 238 of the corresponding image 230 as determined by the LIDAR unit 214. In other words, the depth mapper 218 may populate each pixel within the segment 228 with the depth value 238 of the centroid 230 of the segment. In this approach, the depth map 220 thus includes the depth values ​​238 for the segments 228, where a given segment 228 has a single depth value 238.

[0060] In another approach, the depth mapper 218 can use a weighted average of neighboring fragments 228 to fill each fragment 228 with the depth value 238. For example, the depth mapper 218 can fill the fragments 228 of the same object according to weighted neighboring fragments 228. The weighted average of neighboring fragments 228 can be affected by the color similarity between the fragments 228 and / or the length of their shared borders.

[0061] In another approach, the distance from the centroid 230 may also be a parameter that affects the interpolation of the depth value 238. The depth mapper 218 may create the depth map 220 by merging the fragments 228 with the depth values ​​238 based on the distance from the centroid 230.

[0062] The electronic device 202 may use the depth map 220 for various applications. For example, in one application, the electronic device 102 may use the depth map 220 to build a three-dimensional model of an object as the electronic device 102 moves around the object. In another application, the electronic device 102 may use the depth map 220 for indoor navigation, where the electronic device 102 is mounted on a car driving through a parking lot. The depth map 220 may be used for autofocus applications of the camera 204. Other applications of the depth map 220 may be for creating a map of objects in the scene 222, object detection (e.g., pedestrian detection, traffic sign detection, etc.), autonomous driving, and UAV navigation.

[0063] In an embodiment, the image segmentation mapper 216 may use feedback 240 from a previous depth map 220 to adjust the segments 228 of the image 224. The depth mapper 218 may provide the feedback 240 to the image segmentation mapper 216. The image segmentation mapper 216 may generate more or fewer segments 228 to optimize the segments based on the feedback 240 from the depth mapper 218. For example, if two adjacent segments 228 have similar colors and they have the same depth value 238 in the last 10 frames, the image segmentation mapper 216 may combine the two segments 228 into one segment 228 for the next frame. The LIDAR unit 214 may then take one sample of the combined segment 228 instead of two samples. This may further improve the frame rate and reduce battery consumption.

[0064] like Figure 2 , one or more of the illustrated components may optionally be implemented by the processor 242. In some configurations, different processors may be used to implement different components (e.g., one processor may implement the image segmentation mapper 216, another processor may be used to implement the depth value determination block 236, and another processor may be used to implement the depth mapper 218).

[0065] Figure 3 1 is a flow chart illustrating a method 300 for performing image-assisted LIDAR depth map sampling. The method 300 may be performed as follows: Figure 2 . Specifically, the method 300 may be implemented by the LIDAR unit 214 of the electronic device 202. In some implementations, the method 300 may be partially implemented by the LIDAR unit 214 and a processor (eg, the processor 242) in the electronic device 202.

[0066] refer to Figure 2 and Figure 3 The LIDAR unit 214 can receive 302 the spatial coordinates 232 from the image segmentation mapper 216. The spatial coordinates 232 can be generated based on the image 224 of the scene 222. For example, the image segmentation mapper 216 can receive the image 224 from the camera 204 positioned on or near the LIDAR unit 214. The image segmentation mapper 216 can generate a segmentation map 226 from the image 224. The segmentation map 226 can include a plurality of segments 228. One or more segments 228 can define boundaries of an object in the image 224.

[0067] Each segment 228 may have a centroid 230 and associated spatial coordinates 232. The LIDAR unit 214 may receive 302 the centroid 230 and / or the associated spatial coordinates 232 from the image segmentation mapper 216. In some implementations, the LIDAR unit 214 may receive scanning instructions from a processor, such as the processor 242. The scanning instructions may include commands and / or information regarding positioning, steering, orienting, or otherwise effecting movement of the LIDAR unit 214 to sample the scene 222. In other implementations, the LIDAR unit 214 may receive the segments 228 or the image 224 and generate the segments 228 and / or centroids.

[0068] The LIDAR unit 214 may perform 304 a LIDAR scan based on the spatial coordinates 232. The LIDAR unit 214 may be steered to sample the scene 222 at the spatial coordinates 232. For example, the steerable laser 234 may be steered to take a sample of the scene 222 at a point of the spatial coordinates 232 corresponding to the centroid 230. In another example, the steerable laser 234 may be steered to take a sample of the scene 222 at a point of the spatial coordinates 232 corresponding to the center of the bounding box around the given segment 228. The sampling points may also be determined according to other approaches, such as cluster centers or morphological erosion.

[0069] The LIDAR unit 214 may determine a depth value 238 at the spatial coordinate 232 of the segment 228. The depth value 238 may correspond to a distance of one or more objects from the LIDAR unit 214. In some implementations, the LIDAR unit 214 may provide data, such as temporal data indicative of the depth value, to a processor, such as the processor 242, of the electronic device 202.

[0070] The LIDAR unit 214 may provide 306 the depth values ​​238 obtained through the LIDAR scan to the depth mapper 218 to form the depth map 220. The depth mapper 218 may receive the segments 228 of the image 224 from the image segmentation mapper 216. The depth mapper 218 may fill each segment 228 with the depth value 238 corresponding to the spatial coordinates 232 of the segment 228 provided by the LIDAR unit 214.

[0071] In one approach, the depth mapper 218 may populate each segment 228 with the depth value 238 corresponding to the centroid 230 as determined by the LIDAR unit 214. In another approach, the depth mapper 218 may populate each segment 228 with the depth value 238 using a weighted average of neighboring segments 228. In another approach, the depth mapper 218 may create the depth map 220 by merging the segments 228 with the depth values ​​238 based on the distance from the centroid 230.

[0072] Figure 4 FIG. 4 is a flow chart illustrating another method 400 for performing image-assisted LIDAR depth map sampling. The method 400 may be performed, for example, as Figure 2 The electronic device 202 depicted in FIG.

[0073] refer to Figure 2 and Figure 4 , the electronic device 202 can obtain 402 a color image 224 of the scene 222. For example, the camera 204 can capture the color image 224. The camera 204 can be positioned (e.g., mounted) on or near the LIDAR unit 214. The camera 204 can provide the image 224 to the image segmentation mapper 216.

[0074] The electronic device 202 may generate 404 a segmentation map 226 including a plurality of segments 228 from the image 224. The one or more segments 228 may define the boundaries of an object in the image 224. The number of segments 228 generated by the electronic device 202 may be determined according to different methods. In one approach, the number of segments 228 may be a fixed number. In another approach, the number of segments 228 may be a maximum number. In another approach, the number of segments 228 may vary based on the feedback 240. In another approach, the number of segments 228 may be determined to vary with the image content, wherein a relatively complex scene 222 may utilize a greater number of segments 228 relative to a less complex scene.

[0075] The electronic device 202 may determine 406 the spatial coordinates 232 of the segments 228. The spatial coordinates 232 indicate the location of each segment 228 in the scene 222. The spatial coordinates 232 may be sampling points for performing a LIDAR scan. In one implementation, the electronic device 202 may determine 406 the spatial coordinates 232 of the centroids 230 of the segments 228. Each segment 228 may have a centroid 230. The centroids 230 may be expressed as spatial coordinates 232. For example, the centroid 230 may have an x ​​(e.g., horizontal) coordinate and an x ​​(e.g., vertical) coordinate. The spatial coordinates 232 of the segments 228 may also be determined according to other approaches (e.g., cluster centers or morphological erosion).

[0076] The electronic device 202 may perform 408 a LIDAR scan at the spatial coordinates 232 of the segment 228 to determine the depth value 238. The spatial coordinates 232 may be associated with a sampling point (e.g., a centroid 230, a center of a bounding box, a cluster center, or a morphological erosion) associated with the segment 228. The steerable laser 234 may be steered to take a sample of the scene 222 at a point corresponding to the spatial coordinates 232. For a given spatial coordinate 232 of a segment 228, the electronic device 202 may analyze the amount of time it takes for light to reflect from an object in the scene 222 to determine the depth value 238 for that location.

[0077] The electronic device 202 may merge 410 the depth values ​​238 with the fragments 228 to form the depth map 220 for the scene 222. For example, the electronic device 202 may populate each fragment 228 with the depth value 238 corresponding to the spatial coordinates 232 of the fragment 228 as determined by the LIDAR scan.

[0078] The electronic device 202 may (optionally) use the feedback 240 from the previous depth map 220 to adjust 412 the segments 228 of the image 224. For example, the depth mapper 218 may provide the feedback 240 to the image segmentation mapper 216. The image segmentation mapper 216 may generate more or fewer segments 228 based on the feedback 240 from the depth mapper 218 to optimize the segments.

[0079] Figures 5A to 5C An example of a fragment image 526 generated from an image 524 is illustrated. The image 524 may be a color image captured by a camera 204 located on or near the LIDAR unit 214. The image 524 may be a picture of a scene 222 including one or more objects. The image 524 may be a digital image or may be obtained from a frame of a video.

[0080] In this instance, Figure 5A The image 524 shown in contains multiple objects at different depths. Some of the objects in the image 524 include a cone 554 (on the left side of the image 524), a cup containing pencils 558 (in the right foreground of the image 524), and a grille 560.

[0081] Segmentation map 526 is generated from image 524. Various factors may be considered in determining the number of segments 528 for segmentation of image 524. Factors that may be relevant in determining the number of segments 528 may include image content / complexity, minimum frames per second (fps), desired accuracy (e.g., maximum segment size), and resource utilization (e.g., battery utilization / MIPS / time).

[0082] In this example, segmentation map 526 includes 2000 segments 528. In other examples, fewer or more segments 528 may be utilized. Figure 5B and 5C As can be seen, segments 528 are not uniform in this example, but are irregular in geometry, size, and distribution in response to the content of image 524. The boundaries of segments 528 follow the boundaries or edges of objects from image 524, such as in Figure 5C 528. It should be noted that for thin objects (e.g., pencil 558 in the lower right of image 524), the width of the object can be defined by a single segment 528.

[0083] Each segment 528 has a corresponding centroid 530. In this example, a single centroid 530 is illustrated for a given segment. The centroid 530 may be characterized by spatial coordinates 232. These spatial coordinates 232 may be determined based on the location of the centroid 530 within the image 524. In an embodiment, the centroid 530 may be expressed in terms of an x ​​(e.g., horizontal) coordinate and a (e.g., vertical) coordinate.

[0084] Figures 6A to 6D An example of image-assisted LIDAR depth map 620 sampling is illustrated. Fig. 6A The image 624 depicted in FIG. Figure 5A The same image 524 of the depicted scene. Image 624 includes, among other objects, a plurality of pyramids 654, a pencil 656, and a grid 660. Image 624 may be a digital image or may be obtained from a frame of a video.

[0085] The ground truth 662 is a reference to which the depth map 620 can be compared. The ground truth 662 is contained in Figure 6B The quality of the depth map 620 sampled using the assisted and unassisted approaches, respectively, is visually inspected in FIG. 6. It should be noted that the ground truth 662 is not part of the systems and methods described herein.

[0086] Figure 6C and 6D 6 is a visual representation of the depth values ​​238. The depth values ​​238 are indicated as different shades within the depth map 620. Lighter shades in the depth map 620 indicate that objects are closer to the LIDAR unit 214. Darker shades in the depth map 620 indicate that objects are farther away from the LIDAR unit 214. Thus, objects in the background of the scene are depicted as darker shades in the depth map 620, while objects in the foreground of the scene are depicted as lighter shades in the depth map 620.

[0087] It should be noted that the depth map image is one example visual depiction of depth map information provided for visualization and illustration purposes. In practice, the visualization 620 of depth may be represented differently or not represented at all. Alternatively, the depth map information may be used for various operations or processes, such as building three-dimensional models, indoor navigation, autonomous navigation, object detection, and other applications.

[0088] The unaided LIDAR depth map 620a is Figure 6C 6. The unassisted LIDAR depth map 620a is obtained using raw regular LIDAR sampling of depth values ​​238. In this case, the LIDAR sampling is performed in a uniform manner using 2,000 regularly spaced sampling points. Therefore, this approach does not take into account objects within the scene when determining where to perform the LIDAR scan.

[0089] The image-assisted LIDAR depth map 620b is generated according to the systems and methods described herein. In this case, the segmentation map 226 is created with 2,000 segments 228. The segment 228 boundaries follow the object boundaries, as described in conjunction with FIG. 5. A LIDAR scan is then performed at the spatial coordinates 232 of the segment 228. The depth values ​​238 from the LIDAR scan are then filled in the corresponding segments 228 to generate the image-assisted LIDAR depth map 620b.

[0090] By Figure 6C and 6D As observed by comparison, the unassisted LIDAR depth map 620a produces poor performance, especially at the boundary lines. Figure 6CThe cones 654 in the unassisted LIDAR depth map 620a of FIG. 6 are segmented and the grid 660 opening is misshapen. In addition, thin objects (e.g., pencil 656) may be between the sampling points and may not be detected by the LIDAR scan, as in Figure 6C As demonstrated in .

[0091] LIDAR sampling may use a relatively dense horizontal grid with fairly accurate resolution (e.g., 0.1 degrees). However, in the vertical field of view, the resolution may be fairly coarse (e.g., ranging from 0.5 degrees to 4 degrees). The vertical field of view may be limited to, for example, between 8 and 20 degrees. Scan rates are typically between 10 fps and 50 fps.

[0092] Under these assumptions, one LIDAR scan may be acquired in one frame (ranging from 20ms to 100ms). This is a sufficient interval for the system to account for movement and / or jitter. For example, an inertial measurement unit (IMU) may be used to account for the motion and / or jitter of the LIDAR unit 114. The IMU may be coupled to the LIDAR unit 114 to interpolate self-motion center data (e.g., data related to the three-dimensional motion of the LIDAR unit 114 within the environment) into something referenced in a global coordinate system.

[0093] In embodiments where the LIDAR unit is coupled to a vehicle, the same is true for vehicle motion. By default, the raw data from the LIDAR unit 114 will provide a point cloud in a local coordinate system. Since the described system may be located on a mobile platform (e.g., a vehicle), this motion may be addressed in order to transfer these points to a global reference coordinate system.

[0094] As Fig. 6A , 6B , 6C, and 6D demonstrate that the accuracy of depth map 620b generated by image-assisted LIDAR depth map 620b is better than that of depth map 620a generated by unassisted LIDAR depth map 620a. Although depth map 620 uses the same number of scan points (i.e., 2,000) in this example, it should be noted that depth map 620 generated using fewer scan points according to the image-assisted LIDAR sampling approach may be more accurate than depth map 620 generated using more scan points according to the unassisted approach.

[0095] Since the image-assisted approach takes into account objects in the scene when segmenting the image 624, fewer segments 228 may be required to represent different objects. The segments 228 may be generated to preserve boundaries (e.g., edges) between objects in the image 224. Large segments 228 that preserve boundaries between objects may be sufficient to generate an accurate depth map 620 according to the image-assisted method described herein. Therefore, a depth map 620 generated with a lower segment density may provide sufficient accuracy. In addition, using more segments 228 produces fewer segments 228, which in turn produces fewer scan points. By using fewer LIDAR samples to generate the depth map 620, the electronic device 202 may achieve efficiency gains (e.g., processing, battery consumption, time, etc.) while retaining sufficient accuracy.

[0096] Image-assisted LIDAR depth map 620b is more accurate than unassisted LIDAR depth map 620a. Since segment 228 follows the boundary of the object in image 624, compared with unassisted LIDAR depth map 620a, Fig.6D The image-assisted LIDAR depth map 620b in FIG. 6 retains the sharper edges of objects (e.g., cone 654 and grille 660 openings). In addition, thin objects (e.g., pencil 656) are captured in the image-assisted LIDAR depth map 620b.

[0097] Figures 7A to 7D Another example of image-assisted LIDAR depth map 720 sampling is illustrated. Fig. 7A 724 showing the scene in FIG. Figure 7B 762. The corresponding ground truth 762 is shown in . The image 724 includes, among other objects, fabric 764, plants 766, and a birdhouse 768.

[0098] Figure 7C 7. An unaided LIDAR depth map 720a is illustrated in FIG. The unaided LIDAR depth map 720a is obtained using regular LIDAR sampling of depth values ​​238. In this case, the LIDAR sampling is performed in a uniform manner using 2,000 regularly spaced sampling points. Therefore, this approach does not take into account objects within the scene when determining the location to perform the LIDAR scan.

[0099] The image-assisted LIDAR depth map 720b is generated according to the systems and methods described herein. In this case, the segmentation map 226 is created with 2,000 segments 228. The segment 228 boundaries follow the object boundaries, as described in conjunction with FIG. 5. A LIDAR scan is then performed at the spatial coordinates 732 of the segment 228. The depth values ​​238 from the LIDAR scan are then filled in the corresponding segments 228 to generate the image-assisted LIDAR depth map 720b.

[0100] By Figure 7C and 7D As can be observed from the comparison, the image-assisted LIDAR depth map 720b produces better performance, especially at the object boundary lines. Figure 7C Compared to the unassisted LIDAR depth map 720a, the edges of the fabric 764, the leaves of the plant 766, and the roof of the birdhouse 768 are Fig.7D The image-assisted LIDAR depth map 720b is clearer.

[0101] Figure 8 1 is a block diagram illustrating one example of an electronic device 802 in which systems and methods for image-assisted LIDAR depth map sampling may be implemented. Examples of electronic device 802 include a camera, a video camcorder, a digital camera, a cellular phone, a smart phone, a computer (e.g., a desktop computer, a laptop computer, etc.), a tablet device, a media player, a television, an automobile, a personal camera, an action camera, a surveillance camera, a mounted camera, a tethered camera, a robot, an aircraft, a drone, an unmanned aerial vehicle (UAV), a medical device, a game console, a personal digital assistant (PDA), a set-top box, etc. The electronic device 802 may include one or more components or elements. One or more of the components or elements may be implemented in hardware (e.g., a circuit) or a combination of hardware and software (e.g., a processor with instructions).

[0102] In some configurations, the electronic device 802 may include a processor 842, a memory 844, a display 846, a camera 804, a LIDAR unit 814, and / or a communication interface 805. The processor 842 may be coupled to (e.g., in electronic communication with) the memory 844, the display 846, the camera 804, the LIDAR unit 814, and / or the communication interface 805. Note that Figure 8 One or more of the elements described in the example may be optional. In particular, the electronic device 802 may not include the Figure 8 For example, the electronic device 802 may or may not include the communication interface 805. Additionally or alternatively, the electronic device 802 may or may not include the display 846.

[0103] The communication interface 805 may enable the electronic device 802 to communicate with one or more other electronic devices. For example, the communication interface 805 may provide an interface for wired and / or wireless communications. In some configurations, the communication interface 805 may be coupled to one or more antennas 807 for transmitting and / or receiving radio frequency (RF) signals. Additionally or alternatively, the communication interface 805 may implement one or more wired (e.g., Universal Serial Bus (USB), Ethernet, etc.) communications.

[0104] In some configurations, multiple communication interfaces 805 may be implemented and / or utilized. For example, one communication interface 805 may be a cellular (e.g., 3G, Long Term Evolution (LTE), CDMA, etc.) communication interface 805, another communication interface 805 may be an Ethernet interface, another communication interface 805 may be a Universal Serial Bus (USB) interface, and another communication interface 805 may be a Wireless Local Area Network (WLAN) interface (e.g., an Institute of Electrical and Electronics Engineers (IEEE) 802.11 interface).

[0105] In some configurations, the electronic device 802 may perform a combination of Figures 1 to 8 In addition or alternatively, the electronic device 802 may include one or more of the functions, procedures, methods, steps, etc. described in one or more of the embodiments. Figures 1 to 8 In some configurations, in combination with one or more of the structures described in Figure 8 The described image-assisted LIDAR depth map sampling can be combined with the approach (combined with Figures 1 to 8 It should be noted that, except for the combination Figures 1 to 8 In addition or alternatively to one or more of the image scanning and / or object tracking approaches described in Figure 8 Image-assisted LIDAR depth map sampling as described.

[0106] The electronic device 802 may obtain one or more images 824 (e.g., digital images, image frames, videos, etc.). For example, the electronic device 802 may include a camera 804 having an optical system 106 (e.g., a lens) and an image sensor 108. In some configurations, the camera 804 may capture one or more images 824. The camera 804 may be coupled to and / or controlled by a processor 842. Additionally or alternatively, the electronic device 802 may request and / or receive one or more images 824 from another device (e.g., an external image sensor coupled to the electronic device 802, a network server, a traffic camera, an underwater camera, an automatic camera, a webcam, etc.). In some configurations, the electronic device 802 may request and / or receive one or more images 824 via the communication interface 805. For example, the electronic device 802 may or may not include a camera 804 and may receive images 824 from one or more remote devices.

[0107] One or more images 824 can be stored in memory 844. One or more of images 824 can include objects in scene 222.

[0108] The memory 844 may store instructions and / or data. The processor 842 may access (e.g., read and / or write to) the memory 844. Examples of instructions and / or data that may be stored by the memory 844 may include image data (e.g., one or more images 824), image segmentation mapper 816 instructions, LIDAR unit 814 instructions, depth mapper 818 instructions, and / or instructions for other elements, etc. In some configurations, the electronic device 802 (e.g., the memory 844) may include an image data buffer (not shown). The image data buffer may buffer (e.g., store) image data (e.g., image 824) from the camera 804. The buffered image data may be provided to the processor 842.

[0109] In some configurations, the electronic device 802 may include a camera software application and / or a display 846. While the camera application is running, images 824 of objects within the field of view of the camera 804 may be captured by the camera 804. The images 824 captured by the camera 804 may be presented on the display 846. In some configurations, these images 824 may be displayed in rapid succession at a relatively high frame rate so that at any given etching an object within the field of view of the camera 804 is presented on the display 846. The one or more images 824 obtained by the electronic device 802 may be one or more video frames and / or one or more still images.

[0110] The processor 842 may include and / or implement the image segmentation mapper 816, the LIDAR unit controller 848, and / or the depth mapper 818. In some configurations, the processor 842 may be combined with Figure 2 802 and / or the processor 842. It should be noted that one or more of the elements illustrated in the electronic device 802 and / or the processor 842 may be optional. For example, one or more of the elements illustrated in the processor 842 may or may not be included and / or implemented. Additionally or alternatively, one or more of the elements illustrated in the processor 842 may be implemented separately from the processor 842 (e.g., in other circuits, on another processor, on a separate electronic device, etc.). For example, the LIDAR unit controller 848 may not be implemented on the electronic device 802.

[0111] The processor 842 may include and / or implement the image segmentation mapper 816. In some configurations, the image segmentation mapper 816 may be combined with Figures 1 to 2 224 . The image segmentation mapper 816 can generate a segmentation map 226 from the image 824 of the scene 222. The segmentation map 226 can include a plurality of segments 228.

[0112] The processor 842 may include and / or implement a LIDAR unit controller 848. The LIDAR unit controller 848 may receive the spatial coordinates 230 of the centroid 230 of the segment 228. The LIDAR unit controller 848 may instruct the LIDAR unit 814 to perform a LIDAR scan at the spatial coordinates 232 of the centroid 230. The LIDAR unit 814 may be steered to sample the scene 222 at the spatial coordinates 232. The LIDAR unit 814 may determine the depth value 238 of the scene 222 at the spatial coordinates 232 of the centroid 230.

[0113] The processor 842 may include and / or implement a depth mapper 818. In some configurations, the depth mapper 818 may be an example of one or more of the depth mappers 118, 218 described herein. One or more of the depth values ​​238 may be provided to the depth mapper 818. The depth mapper 818 may populate each segment 228 with the depth value 238 of the centroid 230 of the segment 228 to form the depth map 820.

[0114] In some configurations, the electronic device 802 may perform assisted driving based on the depth map 820. For example, the processor 842 may include (e.g., implement) an advanced driver assistance system (ADAS) or may communicate with it. For example, the electronic device 802 (e.g., ADAS) may perform assisted driving based on the depth map 820. The electronic device 802 may perform one or more operations based on the depth map 820. Examples of operations may include object detection and / or object tracking. Examples of operations may also include displaying indicators (e.g., speed limits, stop signs, pedestrian warnings, possible collision warnings, lane departure warnings, street names, addresses, etc.). Outputting sounds (e.g., honking, sirens, voices, honking vehicle horns, etc.), controlling vehicle speed (e.g., driving at the posted speed limit, braking to avoid a collision, etc.), controlling vehicle steering (e.g., steering to avoid a collision, parallel parking, etc.), controlling vehicle climate (e.g., controlling a defroster or defogger, etc.), controlling vehicle lights (e.g., turning on fog lights, activating emergency flashers, controlling turn signals, etc.). It should be noted that in some configurations, the electronic device 802 may be separate from the automobile or may be integrated into the automobile.

[0115] In some configurations, the processor 842 may include and / or implement one or more other elements. For example, the processor 842 may include an object detector, an object tracker, and the like.

[0116] In some configurations, the electronic device 802 may present a user interface 812 on the display 846. For example, the user interface 812 may enable a user to interact with the electronic device 802. In some configurations, the user interface 846 may be a touch screen 110 that receives input from physical touch (e.g., by a finger, stylus, or other tool). Additionally or alternatively, the electronic device 802 may include or be coupled to another input interface. For example, the electronic device 802 may include a camera facing the user and may detect user gestures (e.g., hand gestures, arm gestures, eye tracking, eyelid blinks, etc.). In another example, the electronic device 802 may be coupled to a mouse and may detect mouse clicks.

[0117] It should be noted that in some configurations no user input may be required. For example, the electronic device 802 may automatically perform image-assisted LIDAR depth map sampling.

[0118] Fig. 9 Certain components that may be included in the electronic device 902 are described. The electronic device 902 may be or may be included in the following: a camera, a video camcorder, a digital camera, a cellular phone, a smart phone, a computer (e.g., a desktop computer, a laptop computer, etc.), a tablet device, a media player, a television, an automobile, a personal camera, an action camera, a surveillance camera, a mounted camera, a connected camera, a robot, an aircraft, a drone, an unmanned aerial vehicle (UAV), a medical device, a game console, a personal digital assistant (PDA), a set-top box, etc. The electronic device 902 includes a processor 942. The processor 942 may be a general-purpose single-chip or multi-chip microprocessor (e.g., an ARM), a special-purpose microprocessor (e.g., a digital signal processor (DSP)), a microcontroller, a programmable gate array, etc. The processor 942 may be referred to as a central processing unit (CPU). Although only a single processor 942 is shown in the electronic device 902, in an alternative configuration, a combination of processors (e.g., an ARM and a DSP) may be used.

[0119] The electronic device 902 also includes memory 944. The memory 944 may be any electronic component capable of storing electronic information. The memory 944 may be embodied as random access memory (RAM), read only memory (ROM), magnetic disk storage media, optical storage media, flash memory devices in RAM, onboard memory included with the processor, EPROM memory, EEPROM memory, registers, etc., including combinations thereof.

[0120] The data 950a and instructions 952a may be stored in the memory 944. The instructions 952a may be executed by the processor 942 to implement one or more of the methods described herein. Executing the instructions 952a may involve using the data stored in the memory 944. When the processor 942 executes the instructions 952, various portions of the instructions 952b may be loaded onto the processor 942, and various data 950b segments may be loaded onto the processor 942.

[0121] The electronic device 902 may also include a transmitter 925 and a receiver 927 to allow transmission and reception of signals to and from the electronic device 902. The transmitter 925 and the receiver 927 may be collectively referred to as a transceiver 935. One or more antennas 907a-b may be electrically coupled to the transceiver 935. The electronic device 902 may also include (not shown) multiple transmitters, multiple receivers, multiple transceivers, and / or additional antennas.

[0122] The electronic device 902 may include a digital signal processor (DSP) 931. The electronic device 902 may also include a communication interface 905. The communication interface 905 may allow one or more types of input and / or output to be implemented. For example, the communication interface 905 may include one or more ports and / or communication devices for linking other devices to the electronic device 902. Additionally or alternatively, the communication interface 905 may include one or more other interfaces (e.g., a touch screen, a keypad, a keyboard, a microphone, a camera, etc.). For example, the communication interface 905 may enable a user to interact with the electronic device 902.

[0123] The various components of the electronic device 902 may be coupled together via one or more buses, which may include a power bus, a control signal bus, a status signal bus, a data bus, etc. For clarity, the various buses are described in detail in FIG. Fig. 9 It is described as a bus system 923.

[0124] According to the present invention, a circuit in an electronic device may be adapted to obtain spatial coordinates received from an image segmentation mapper. The spatial coordinates may be generated based on an image of a scene. The same circuit, a different circuit, or a second section of the same or different circuit may be adapted to perform vertical processing of a depth map to perform a LIDAR scan. The LIDAR unit may be turned to sample the scene at the spatial coordinates. The same circuit, a different circuit, or a third section of the same or different circuit may be adapted to provide depth values ​​obtained by the LIDAR scan to a depth mapper to form a depth map. Additionally, the same circuit, a different circuit, or a fourth section of the same or different circuit may be adapted to control the configuration of current or circuit sections that provide the functionality described above.

[0125] The term "determining" encompasses a wide variety of actions, and thus "determining" may include computing, calculating, processing, deriving, investigating, looking up (e.g., looking up in a table, a database, or another data structure), determining, etc. Additionally, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Furthermore, "determining" may include resolving, selecting, choosing, establishing, etc.

[0126] Unless expressly specified otherwise, the phrase “based on” does not mean “based only on.” In other words, the phrase “based on” describes both “based only on” and “based at least on.”

[0127] The term "processor" should be broadly disclosed to encompass general purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc. In some cases, a "processor" may refer to an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. The term "processor" may refer to a combination of processing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0128] The term "memory" should be interpreted broadly to encompass any electronic component capable of storing electronic information. The term memory may refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), nonvolatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, etc. A memory is considered to be in electronic communication with a processor if the processor can read information from and / or write information to the memory. Memory that is integral to a processor is in electronic communication with the processor.

[0129] The terms "instructions" and "code" should be broadly disclosed to include any type of computer-readable statements. For example, the terms "instructions" and "code" may refer to one or more procedures, routines, subroutines, functions, programs, etc. "Instructions" and "code" may include a single computer-readable statement or many computer-readable statements.

[0130] The functions described herein may be implemented by software or firmware executed by hardware. The functions may be stored as one or more instructions on a computer-readable medium. The term "computer-readable medium" or "computer program product" refers to any tangible storage medium that can be accessed by a computer or a processor. By way of example and not limitation, computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or other media that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer. As used herein, disks and optical disks include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray disks. Optical disks, where magnetic disks typically reproduce data magnetically, and optical disks reproduce data optically by means of lasers. It should be noted that computer readable media may be tangible and non-volatile. The term "computer program product" refers to a computing device or processor in combination with code or instructions (e.g., a "program") that can be executed, processed, or computed by a computing device or processor. As used herein, the term "code" may refer to software, instructions, code, or data that can be executed by a computing device or processor.

[0131] The software or instructions may also be transmitted via a transmission media. For example, if the software is transmitted from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave are included in the definition of media.

[0132] The methods disclosed herein include one or more steps or actions for implementing the described methods. The method steps and / or actions may be interchangeable with each other without departing from the scope of the claims. In other words, unless the proper operation of the described method requires a specific order of steps or actions, the order and / or use of the specific steps and / or actions may be modified without departing from the scope of the claims.

[0133] In addition, it should be understood that modules and / or other appropriate devices for performing the methods and techniques described herein may be downloaded and / or otherwise obtained by the device. For example, the device may be coupled to a server to facilitate the transfer of devices for performing the methods described herein. Alternatively, the various methods described herein may be provided via a storage device (e.g., random access memory (RAM), read-only memory (ROM), physical storage media such as a compact disk (CD) or a floppy disk, etc.), so that the device can obtain the various methods when the storage device is coupled or provided to the device.

[0134] It is to be understood that the claims are not limited to the precise configuration and components described above. Various modifications, changes and variations may be made in the arrangement, operation and details of the systems, methods and apparatus described herein without departing from the scope of the claims.

Claims

1. A device for automatic control of a vehicle, comprising: Memory; as well as at least one processor coupled to the memory, the at least one processor being configured to: generating a first plurality of sets of spatial coordinates based on a content analysis of a first received image, each set of spatial coordinates indicating a location of a segment within a first object in the first received image, wherein at least a portion of the segment has a boundary that follows a boundary of the first object; providing the first plurality of sets of spatial coordinates to a LIDAR unit communicatively coupled to the processor to selectively obtain information indicative of depths at locations in a scene external to the vehicle representative of the first plurality of spatial coordinates; determining a position of the first object relative to the vehicle based on the first plurality of sets of spatial coordinates and the depth information received from the LIDAR unit; and Instructions for vehicle operation are determined based on the determined position of the first object.

2. The device according to claim 1, wherein: At least a portion of the segments are irregular in geometry and size, and the segments are unevenly distributed within the first object.

3. The device according to claim 1, wherein: The processor is further configured to: generating a second plurality of spatial coordinate sets for the first object based on the received depth information and the first plurality of spatial coordinate sets, wherein the second plurality of spatial coordinate sets are distributed differently within the first object relative to the first plurality of spatial coordinate sets; and The second plurality of sets of spatial coordinates are provided to the LIDAR unit to selectively obtain updated information indicative of depths at locations in the scene external to the vehicle that are representative of the second plurality of spatial coordinates.

4. The device according to claim 3, wherein: The first plurality of sets of spatial coordinates and the received depth information are transformed based on movement of the vehicle.

5. The device according to claim 3, wherein: The content analysis includes the processor being configured to perform at least one of the following operations: determining a plurality of boundaries of the first object; or determining a color or grayscale difference of the first object.

6. The device according to claim 5, wherein: At least a portion of the plurality of boundaries has an irregular geometry that aligns with one or more of the objects.

7. The device according to claim 1, wherein: The processor or the LIDAR unit is configured to convert the first plurality of sets of spatial coordinates into a LIDAR coordinate space.

8. The device according to claim 1, wherein: The first object is one of a second vehicle, a pedestrian, a lane boundary, or a sign.

9. The device according to claim 1, wherein: The vehicle operation is at least one of a speed change, a braking operation, a steering operation, or a driver or passenger notification.

10. A method for automatic control of a vehicle, comprising: generating a first plurality of sets of spatial coordinates based on a content analysis of a first received image, each set of spatial coordinates indicating a location of a segment within a first object in the first received image, wherein at least a portion of the segment has a boundary that follows a boundary of the first object; providing the first plurality of sets of spatial coordinates to a LIDAR unit to selectively obtain information indicating depths at locations in a scene external to the vehicle that are representative of the first plurality of spatial coordinates; determining a position of the first object relative to the vehicle based on the first plurality of sets of spatial coordinates and the depth information received from the LIDAR unit; and Instructions for vehicle operation are determined based on the determined position of the first object.

11. The method according to claim 10, wherein: At least a portion of the segments are irregular in geometry and size, and the segments are unevenly distributed within the first object.

12. The method according to claim 10, further comprising: generating a second plurality of spatial coordinate sets for the first object based on the received depth information and the first plurality of spatial coordinate sets, wherein the second plurality of spatial coordinate sets are distributed differently in the first object relative to the first plurality of spatial coordinate sets; and The second plurality of sets of spatial coordinates are provided to the LIDAR unit to selectively obtain updated information indicative of depths at locations in the scene external to the vehicle that are representative of the second plurality of spatial coordinates.

13. The method according to claim 10, wherein: The content analysis includes at least one of the following operations: determining a plurality of boundaries for the first object; or determining a color or grayscale difference for the first object.

14. The method according to claim 13, wherein: At least a portion of the plurality of boundaries has an irregular geometry that aligns with one or more of the objects.

15. An imaging device comprising: a camera configured to obtain image information of a scene, wherein the camera is capable of automatically focusing on an area of ​​interest in the scene; a LIDAR unit configured to obtain depth information of at least a portion of the scene; and A processor configured to: detecting an object in the scene using image information obtained from the camera and first depth information obtained from the LIDAR unit; providing one or more instructions to the LIDAR unit to obtain second depth information at a plurality of specified scanning locations, wherein the specified scanning locations are characterized by: a first scanning point density for a first portion of the scene associated with the object, and a second scanning point density for a second portion of the scene; and One or more instructions are provided to the camera based on the obtained second depth information associated with the designated scanning position associated with the object to adjust the focus of the camera to the area of ​​interest and capture updated image information of the scene.

16. The imaging device according to claim 15, wherein The processor is further configured to generate a first image segment of the image information associated with the object; and generate a second image segment of the image information associated with a portion of the scene not associated with the object.

17. The imaging device according to claim 16, wherein: The specified scanning position of the first portion of the scene is based on the first image segment, and wherein the specified scanning position of the second portion of the scene is based on the second image segment.

18. The imaging device according to claim 17, wherein: The first scanning point density is greater than the second scanning point density.

19. The imaging device according to claim 15, wherein: The object is a person or a face.

20. The imaging device according to claim 19, wherein The processor is also configured to detect the face using facial recognition.

21. The imaging device according to claim 15, wherein The processor is further configured to generate segments of the image information, and wherein the first scanning point density is based on similarity of image information of adjacent image segments within the object and similarity of first depth information of adjacent image segments within the object.

22. The imaging device according to claim 15, further comprising a user interface configured to display image information obtained from the camera and to receive a selection of the object input by a user, wherein: The processor is further configured to detect the object based on selection information received from the user interface.

23. The imaging device according to claim 15, wherein: The processor is further configured to generate a depth map for at least a portion of the scene based on depth information obtained from the LIDAR unit.

24. The imaging device according to claim 23, wherein The processor is further configured to build a three-dimensional model of the object based on the depth map.

25. The imaging device according to claim 23, wherein The processor is further configured to: generate image segments of the image information associated with the object; and generate a refined depth map based on the image segments.

26. A method for capturing an image using an electronic device, comprising: Obtaining image information of a scene from a camera of the electronic device; Obtaining depth information of at least a portion of the scene using a LIDAR unit of the electronic device; detecting, by a processor, objects in the scene using the image information obtained from the camera and the depth information obtained from the LIDAR unit; providing, by the processor, a plurality of designated scanning positions to the LIDAR unit to obtain updated depth information, wherein the designated scanning positions are characterized by: a first scanning point density for a first portion of the scene associated with the object, and a second scanning point density for a second portion of the scene; providing one or more instructions to the camera based on the updated depth information associated with the designated scanning location associated with the object to adjust the focus of the camera to an area of ​​interest associated with the object; and Updated image information of the scene is captured using the adjusted focus.

27. The method according to claim 26, further comprising: generating a first image segment of the image information associated with the object; and generating a second image segment of the image information associated with a portion of the scene not associated with the object.

28. The method according to claim 27, wherein: The specified scanning position of the first portion of the scene is based on the first image segment, and wherein the specified scanning position of the second portion of the scene is based on the second image segment.

29. The method according to claim 28, wherein: The first scanning point density is greater than the second scanning point density.

30. The method of claim 26, wherein: The object is a person or a face.

31. The method of claim 30, further comprising: The face is detected by the processor using facial recognition.

32. The method of claim 26, further comprising generating, by the processor, segments of the image information, and wherein: The first scanning point density is based on similarity of image information of adjacent image segments within the object and similarity of first depth information of adjacent image segments within the object.

33. The method of claim 26, further comprising: displaying image information obtained from the camera on a screen; as well as Using a user interface, object selection information is obtained based on the displayed image information, wherein detecting the object by the processor is based on the object selection information.

Images