Photosensitive chip capable of realizing fusion of lidar point cloud and image

Abstract

The present application relates to the technical field of laser radar depth information and image information fusion, and particularly relates to a photosensitive chip capable of realizing laser radar point cloud and image fusion, comprising a plurality of matrix-distributed photosensitive pixel units, the photosensitive pixel unit comprising a plurality of matrix-distributed visible light photosensitive pixels and laser photosensitive pixels, the laser photosensitive pixels being inlaid between the plurality of matrix-distributed visible light photosensitive pixels, the laser photosensitive pixels providing depth detection information for the surrounding visible light photosensitive pixels, and the visible light photosensitive pixels providing object feature information for the laser photosensitive pixels. Compared with the prior art, the photosensitive chip capable of realizing laser radar point cloud and image fusion of the present application realizes the fusion of depth measurement and image, can simplify the system structure, improve the stability of the system, can realize the real-time fusion of the depth information measured by the laser photosensitive pixels and the color information measured by the visible light pixels, and avoid complex fusion algorithms.

Description

[Technical Field]

[0001] This invention relates to the field of fusion technology of lidar depth information and image information, and particularly to a photosensitive chip that can realize the fusion of lidar point cloud and image. [Background Technology]

[0002] LiDAR can detect the three-dimensional information of an object and generate a depth point cloud map of the object. The image information of the object provides feature information such as texture and color, which is convenient for identification. By fusing the depth information of the LiDAR with the feature information of the image, a color three-dimensional image is formed, which is beneficial for the recognition, processing and application of three-dimensional images.

[0003] Existing fusion solutions use two independent component systems: LiDAR and camera. This results in complex structural design, high stability requirements, and the need to calibrate both the LiDAR and camera image systems before acquiring point clouds and images of the object under test. The image fusion algorithm is complex, and the two systems also increase costs. [Summary of the Invention]

[0004] To overcome the above problems, this invention proposes a photosensitive chip that can effectively solve the above problems and realize the fusion of lidar point clouds and images.

[0005] The present invention provides a technical solution to solve the above-mentioned technical problems by providing a photosensitive chip that can realize the fusion of lidar point clouds and images, including multiple photosensitive pixel units arranged in a matrix. Each photosensitive pixel unit includes multiple visible light photosensitive pixels and laser photosensitive pixels arranged in a matrix. The laser photosensitive pixels are embedded among the multiple visible light photosensitive pixels arranged in a matrix. The laser photosensitive pixels provide depth detection information to the surrounding visible light photosensitive pixels, and the visible light photosensitive pixels provide object feature information to the laser photosensitive pixels.

[0006] Preferably, the visible light photosensitive pixel includes a visible light photosensitive area and a laser photosensitive area, and the laser photosensitive pixel is disposed in the laser photosensitive area.

[0007] Preferably, the number of laser photosensitive pixels accounts for 10% of the sum of the number of visible light photosensitive pixels and the number of laser photosensitive pixels.

[0008] Preferably, the number of laser photosensitive pixels accounts for 20% of the sum of the number of visible light photosensitive pixels and the number of laser photosensitive pixels.

[0009] Preferably, the number of laser photosensitive pixels accounts for 30% of the sum of the number of visible light photosensitive pixels and the number of laser photosensitive pixels.

[0010] Preferably, the number of laser photosensitive pixels accounts for N percent of the sum of the number of visible light photosensitive pixels and the number of laser photosensitive pixels, where N is greater than zero and less than or equal to fifty percent.

[0011] Compared with existing technologies, the photosensitive chip of the present invention, which enables the fusion of lidar point clouds and images, integrates lidar ranging and image testing chips onto the same chip, realizing the fusion of depth measurement and images. This simplifies the system structure and improves the system stability. By interpolating, the depth information measured by the lidar photosensitive pixels and the color information measured by the visible light pixels can be fused in real time, avoiding complex fusion algorithms. [Attached Image Description]

[0012] Figure 1 This is a schematic diagram of a conventional 4000*3000 visible light CMOS chip structure in the prior art;

[0013] Figure 2 A schematic diagram illustrating the calculation of signals other than the color signal received by each pixel of a conventional 4000*3000 visible light CMOS chip in the prior art using an interpolation algorithm;

[0014] Figure 3 This is a schematic diagram of the region division of the photosensitive chip of the present invention, which enables the fusion of lidar point clouds and images.

[0015] Figure 4 This is a schematic diagram of the first embodiment of the photosensitive chip of the present invention that can realize the fusion of lidar point cloud and image;

[0016] Figure 5 for Figure 4 Enlarged view of a single photosensitive pixel unit;

[0017] Figure 6 This is a schematic diagram of a second embodiment of the photosensitive chip of the present invention that can realize the fusion of lidar point clouds and images;

[0018] Figure 7 for Figure 6 A magnified view of a single photosensitive pixel unit.

Detailed Implementation Methods

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0020] It should be noted that in the embodiments of the present invention, all directional indications (such as up, down, left, right, front, back, etc.) are limited to relative positions on the specified view, rather than absolute positions.

[0021] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0022] Please see Figure 1 and Figure 2 The structure of a typical 4000*3000 visible light CMOS chip is shown in the figure. For this Bayer mode sensor, each R (red light-sensing pixel), G (green light-sensing pixel), and B (blue light-sensing pixel) pixel can only receive its corresponding red, green, and blue monochromatic light. To recover a color image, an interpolation algorithm is needed for each pixel to obtain signals other than the color signal it receives. For example, for... Figure 2 The B6 pixel channel needs to be interpolated to obtain red and green information, for example:

[0023] R color information is obtained by averaging the values ​​of four adjacent R pixels:

[0024] R B6 = (R1+R3+R9+R11) / 4

[0025] G color information is obtained by averaging the values ​​of four adjacent R pixels:

[0026] G B6 = (G2+G5+G7+G10) / 4

[0027] Conventional visible light sensors can collect texture, color, and other feature information of an object by receiving visible light reflected or emitted by the object, but cannot detect depth information; LiDAR collects depth information of an object by emitting infrared laser light onto it and receiving the laser signal reflected by the object, but cannot collect color information of the object.

[0028] Inspired by this, please refer to the following: Figures 3 to 7 The present invention discloses a photosensitive chip capable of fusing lidar point clouds and images, comprising multiple photosensitive pixel units arranged in a matrix. Each photosensitive pixel unit includes multiple visible light photosensitive pixels and laser photosensitive pixels arranged in a matrix. The laser photosensitive pixels are embedded among the multiple visible light photosensitive pixels arranged in a matrix. The laser photosensitive pixels provide depth detection information to the surrounding visible light photosensitive pixels, and the visible light photosensitive pixels provide object feature information to the laser photosensitive pixels. The visible light photosensitive pixels include three types of pixels: R (red light photosensitive pixels), G (green light photosensitive pixels), and B (blue light photosensitive pixels).

[0029] The number of laser photosensitive pixels accounts for 10% of the sum of the number of visible light photosensitive pixels and the number of laser photosensitive pixels.

[0030] In the first embodiment, for a 4000*3000 photosensitive chip, it is divided into 100 regions of 400*300 pixels each. A laser photosensitive pixel X is inserted at the center pixel position of each region. The distance information measured by the laser photosensitive pixel X can be copied to other visible light photosensitive pixels in this 400*300 region. At the same time, the R, G, and B information of the laser photosensitive pixel X can be obtained by interpolation, for example:

[0031] Rx=(R1+R2+R3+R4+R5+R6+R7+R8) / 8

[0032] Gx=(G1+G2+G3+G4+G5+G6+G7+G8) / 8

[0033] Bx=(B1+B2+B3+B4+B5+B6+B7+B8) / 8

[0034] After obtaining Rx, Gx, and Bx, the remaining visible light photosensitive pixels can obtain R, G, and B information through the aforementioned interpolation method.

[0035] Furthermore, the number of laser-sensing pixels can also account for 20%, 30%, or N% of the sum of visible light-sensing pixels and laser-sensing pixels, where N is greater than zero and less than or equal to 50%. The insertion method is similar to the method described above. The higher the proportion of laser-sensing pixels inserted, the more accurate the distance information obtained, and the closer it is to the true value. However, as the number of inserted laser-sensing pixels increases, the number of visible light-sensing pixels decreases, resulting in less collected color information and reduced color reproduction. Therefore, an appropriate insertion ratio needs to be selected based on different testing requirements.

[0036] In the second embodiment, the photosensitive chip includes multiple photosensitive pixel units arranged in a matrix. Each photosensitive pixel unit includes multiple visible light photosensitive pixels arranged in a matrix. Each visible light photosensitive pixel includes a visible light photosensitive area and a laser photosensitive area, with laser photosensitive pixels disposed within the laser photosensitive area. Each laser photosensitive area acquires distance information, and the visible light photosensitive area acquires color information, ensuring that each photosensitive pixel unit can simultaneously provide distance and color information, which helps improve the accuracy of visible light images and distance detection.

[0037] The photosensitive chip of the present invention, which enables the fusion of lidar point clouds and images, can be used in solid-state lasers or in hybrid solid-state and mechanical lidar systems. Using the photosensitive chip of the present invention, which enables the fusion of lidar point clouds and images, depth and image information detection can be achieved in hybrid solid-state lidar.

[0038] Compared with existing technologies, the photosensitive chip of the present invention, which enables the fusion of lidar point clouds and images, integrates lidar ranging and image testing chips onto the same chip, realizing the fusion of depth measurement and images. This simplifies the system structure and improves the system stability. By interpolating, the depth information measured by the lidar photosensitive pixels and the color information measured by the visible light pixels can be fused in real time, avoiding complex fusion algorithms.

[0039] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any modifications, equivalent substitutions and improvements made within the concept of the present invention should be included within the patent protection scope of the present invention.

Claims

1. A photosensitive chip capable of fusing lidar point clouds with images, characterized in that, It includes multiple photosensitive pixel units arranged in a matrix. Each photosensitive pixel unit includes multiple visible light photosensitive pixels and laser photosensitive pixels arranged in a matrix. The laser photosensitive pixels are embedded among the multiple visible light photosensitive pixels arranged in a matrix. The laser photosensitive pixels provide depth detection information to the surrounding visible light photosensitive pixels, and the visible light photosensitive pixels provide object feature information to the laser photosensitive pixels. The visible light photosensitive pixels include red light photosensitive pixels R, green light photosensitive pixels G, and blue light photosensitive pixels B; The number of laser photosensitive pixels accounts for N percent of the sum of the number of visible light photosensitive pixels and the number of laser photosensitive pixels, where N is greater than zero and less than or equal to thirty. The photosensitive surface of the photosensitive chip is divided into multiple regular regions. A laser photosensitive pixel is inserted at the center of each regular region. The distance information measured by the laser photosensitive pixel can be copied to other visible light photosensitive pixels in the same region. The R, G, and B information of the laser photosensitive pixel can be obtained by eight-neighborhood interpolation. The insertion ratio is determined according to different testing requirements. The accuracy of distance information can be adjusted by changing the ratio of the number of laser photosensitive pixels to the sum of the number of visible light photosensitive pixels and laser photosensitive pixels.

2. The photosensitive chip capable of fusing lidar point clouds and images as described in claim 1, characterized in that, The number of laser photosensitive pixels accounts for 10% of the sum of the number of visible light photosensitive pixels and the number of laser photosensitive pixels.

3. The photosensitive chip capable of fusing lidar point clouds and images as described in claim 1, characterized in that, The number of laser photosensitive pixels accounts for 20% of the sum of the number of visible light photosensitive pixels and the number of laser photosensitive pixels.

4. The photosensitive chip capable of fusing lidar point clouds and images as described in claim 1, characterized in that, The number of laser photosensitive pixels accounts for 30% of the sum of the number of visible light photosensitive pixels and the number of laser photosensitive pixels.

5. A photosensitive chip capable of fusing lidar point clouds with images, characterized in that, It includes multiple photosensitive pixel units arranged in a matrix, each photosensitive pixel unit including multiple visible light photosensitive pixels and laser photosensitive pixels arranged in a matrix; each visible light photosensitive pixel includes a visible light photosensitive area and a laser photosensitive area, and a laser photosensitive pixel is disposed in the laser photosensitive area; the laser photosensitive area acquires distance information, and the visible light photosensitive area acquires color information, so that each photosensitive pixel unit can simultaneously provide distance information and color information.

Images