An apparatus for implementing a 3D point cloud coordinate calculation system on an image sensor chip

Abstract

The present invention discloses a device for implementing a 3D point cloud coordinate calculation system on an image sensor chip, characterized in that the device consists of a 3D image sensor chip, a 3D point cloud generation unit, a sensor array, an active light source, and an optical lens mechanism. A pixel memory array, a row recorder, and a column recorder are designed on the 3D image sensor chip. When the active light source emits light to the object to be measured, the row selection and column selection are controlled to read the exposure information of each pixel of the image sensor while recording the valid row selection and column selection addresses of the pixel, thereby generating a coordinate pixel unit, that is, pixel position coordinate information. The 3D point cloud generation unit calculates the 3D point cloud coordinates of the object to be measured based on the obtained coordinate pixel unit and the pre-calibrated center point parameters of the light source and the sensor lens.

Description

Technical Field

[0001] The present invention belongs to the technical fields of machine vision, artificial intelligence, and 3D sensor equipment. Background Art

[0002] Currently, the design and implementation method of 3D cameras is to use one or two 2D cameras to complete imaging, then calculate the coordinate positions of each pixel, and finally calculate the 3D point cloud coordinates of the object to be measured. Its defects and deficiencies are as follows: 1) It consumes a large amount of computing resources. Since the 2D camera needs to generate a 2D image visible to the human eye after sensor imaging and ISP image signal processing, and when generating the 3D point cloud, it mainly locates the coordinates of each pixel and does not require the 2D image visible to the human eye; 2) Poor real-time performance, and the current highest performance is 2 frames per second; 3) There are many limitations in imaging objects to be measured with high-reflective objects, black low-reflective objects, surface oil stains, and complex strong ambient light. Since it is implemented with a 2D camera, the RGGB filter structure of the 2D camera sensor cannot effectively filter out the light that is useless for generating the 3D point cloud; 4) High power consumption, and a large amount of computing resources are required for 2D camera ISP processing.

[0003] It can be seen from this that to enable the 3D camera to solve the above problems, it is necessary to innovate from the level of the image sensor integrated circuit to solve the above problems. Considering directly obtaining the coordinate information (x, y) of the pixel while the row selection and column selection of the image sensor chip are effective and reading the pixel voltage value after the pixel of the image sensor is exposed for subsequent use in generating the 3D point cloud. Summary of the Invention

[0004] The present invention discloses a device for implementing a 3D point cloud coordinate calculation system on an image sensor chip. By designing a 3D image sensor chip, a pixel memory array, and a row and column address recorder, when the active light source emits light to the object to be measured, the pixel memory array reads the exposure information of each pixel of the sensor and records the row and column address information of the pixel at the same time to form a coordinate pixel unit. The 3D point cloud generation unit calculates the 3D point cloud coordinates of the object to be measured according to the obtained position coordinate information of the coordinate pixel unit and the pre-calibrated center point parameters of the light source and the sensor lens. The advantage of the present invention is that it can quickly calculate the 3D point cloud coordinates on a CMOS chip.

[0005] To achieve the above object, the technical solution of the present invention is realized as follows:

[0006] An apparatus for implementing a 3D point cloud coordinate calculation system on an image sensor chip, the apparatus consists of a 3D image sensor chip, a 3D point cloud generation unit, a sensor array, an active light source and an optical lens mechanism. A pixel memory array, a row recorder and a column recorder are designed on the 3D image sensor chip. When the active light source emits light to the object to be measured, while controlling the reading of the exposure information of each pixel of the image sensor through row selection and column selection, the valid row selection and column selection addresses of the pixel are recorded, and then a coordinate pixel unit is generated, that is, the pixel position coordinate information; the 3D point cloud generation unit calculates the 3D point cloud coordinates of the object to be measured according to the obtained coordinate pixel unit and the pre-calibrated center point parameters of the light source and the sensor lens.

[0007] The apparatus further includes the following steps:

[0008] a. Control the active light source to emit light and project a ray of light onto the object to be measured;

[0009] b. The 3D image sensor controls pixel exposure;

[0010] c. Select rows and column selection to control and lock the pixels to be read;

[0011] d. Perform ADC analog-to-digital conversion on the pixels to be read;

[0012] e. The pixel memory array sends the coordinate pixel unit formed by pixel data and pixel coordinates to the 3D point cloud generation unit;

[0013] f. The 3D coordinate generation unit uses the row address and column address of the coordinate pixel, according to the integrated circuit process node implemented by the sensor array chip and the size parameters of each pixel, calculates the coordinates x, y of the pixel, the center point of the imaging lens of the image sensor, the distance parameter between the sensor and the active light source, and the equation X / x = Y / y = Z / f to calculate the 3D point cloud coordinates X, Y, Z; where: f is the focal length value of the lens.

[0014] g. The object to be measured moves for imaging, and steps a to g are repeated until the generation of the 3D point cloud coordinates of the object to be measured is completed;

[0015] Preferably, the ADC analog-to-digital conversion method is that each row of pixels is provided with a unique analog-to-digital conversion unit, which can achieve high-speed parallel analog-to-digital conversion of a row of pixels.

[0016] Preferably, the ADC analog-to-digital conversion method is that each pixel is provided with a unique analog-to-digital conversion unit, which can achieve high-speed parallel analog-to-digital conversion of all pixels.

[0017] Further, the pixel memory array of the 3D image sensor chip integrates the pixel data information read from the DAC analog-to-digital conversion unit with the row address and column address information of the selected row and column of the read pixel into a coordinate pixel unit. The coordinate pixel unit includes: an a-bit binary row address, a b-bit binary column address, and pixel data information. Among them, 2 a is the total number of pixels in one row of the sensor pixel array, 2 b is the total number of pixels in one column of the pixel array.

[0018] Preferably, to reduce the workload of subsequent transmission and processing, the pixel memory array filters and judges the read pixel information, eliminates the pixel data that fails to be exposed and imaged, and only sends the pixel information and coordinates that can be exposed and imaged to the 3D point cloud generation unit.

[0019] Further, the design methods of the pixel memory array include but are not limited to: 1) reading one pixel serially to generate a coordinate pixel unit; 2) reading one row of pixels in parallel to generate one row of coordinate pixel units; 3) reading the global coordinate pixel unit simultaneously.

[0020] Further, the active light source includes: a line laser emitter light source, an LED light source; to ensure the generation of the 3D coordinates of the whole measured object, the imaging method of the measured object moving includes: the measured object moves autonomously, the 3D point cloud coordinate generation device moves autonomously, and the light emitted by the line laser is moved by a laser galvanometer.

[0021] After the 3D image sensor completes the acquisition of pixel and its coordinate data, it is transmitted to the 3D point cloud generation unit to calculate the 3D point cloud coordinates of the measured object. The method includes the following steps:

[0022] a. Calculate the actual coordinate values (x, y) of the pixel according to the actual size of the physical pixel and the row and column coordinate information where the pixel is located;

[0023] b. Use the principle of small hole imaging at the center point of the image of the optical lens to establish an equation between the actual coordinate values of the pixel and the 3D coordinates (X, Y, Z) of the measured object; X / x = Y / y = Z / f, where: f is the focal length of the optical lens of the 3D sensor device;

[0024] c. Calculate the 3D coordinates (X, Y, Z) of the corresponding imaging pixel of the measured object by using the known distance data between the 3D image sensor and the active light source and the plane equation formed by the intersection of the light source ray and the measured object in space.

[0025] Further, to resist ambient light interference, the sensor pixel array includes a physical sensor and a light filtering material. The light filtering material includes: a light filtering material with the same wavelength as the light emitted by the active light source;

[0026] The physical sensor includes: a CMOS (Complementary Metal Oxide Semiconductor) image sensor and a CCD (Charge Couple Device) sensor.

[0027] Furthermore, the actual implementation methods of the 3D point cloud generation unit include: 1) Field Programmable Gate Array (FPGA); 2) Central Processing Unit (CPU) + Graphics Processing Unit (GPU); 3) System on Chip (SoC); 4) Integrated in the 3D image sensor chip and using the same integrated circuit process as the 3D image sensor.

[0028] Furthermore, the device configurations that can be achieved by the 3D image sensor chip at least include: 1) 3D area array camera; 2) 3D line scan camera; 3) 3D lidar device. Description of the Drawings

[0029] Figure 1 For applying the system principle of the present invention

[0030] Figure 2 Schematic diagram of the principle of the 3D image sensor chip for applying the present invention

[0031] Figure 3 Example of the working process of the chip for applying the present invention

[0032] Figure 4 First embodiment for applying the present invention

[0033] Figure 5 Second embodiment for applying the present invention

[0034] Figure 6 Third embodiment for applying the present invention

[0035] Figure 7 Fourth embodiment for applying the present invention Detailed Description of the Invention

[0036] The idea of the present invention is:

[0037] Aiming at the problems of rapid generation of 3D point cloud coordinates, anti-environmental light interference, and complex design of 3D cameras, the present invention discloses a device for directly obtaining pixel coordinates on an image sensor chip to achieve the generation of 3D point cloud coordinates. By designing a 3D image sensor chip, a pixel memory array is designed inside the image sensor chip. When the active light source emits light to the object to be measured, while the pixel memory array reads the exposure information of each pixel of the sensor, it records the position coordinate information of the pixel to form a coordinate pixel unit. After directly obtaining the coordinate information of the pixel, the 3D point cloud generation unit calculates the 3D point cloud coordinates of the object to be measured according to the obtained pixel position coordinate information and the pre-calibrated parameters of the center points of the light source and the sensor lens.

[0038] Figure 1 The system schematic diagram applying the present invention is shown as follows.

[0039] It is composed of a 3D image sensor chip, a 3D point cloud generation unit, a sensor array, an active light source, and an optical lens mechanism.

[0040] Its working principle is as follows: The 3D image sensor chip designs a pixel memory array. When the active light source emits light to the object to be measured, while controlling the row selection and column selection to read the exposure information of each pixel of the sensor, the effective positions of the row selection and column selection of this pixel are recorded as coordinate information. The 3D point cloud generation unit calculates the 3D point cloud coordinates of the object to be measured according to the obtained pixel position coordinate information and the pre-calibrated center point parameters of the light source and the sensor lens.

[0041] Figure 2 The schematic diagram of the principle of the 3D image sensor chip applying the present invention is shown as follows.

[0042] In the image sensor chip, a pixel memory array, a row address recorder, and a column address recorder are designed. While reading the pixels of the sensor, the row address and column address of this pixel are recorded to generate a coordinate pixel unit. The coordinate pixel unit includes: an a-bit binary row address, a b-bit binary column address, and pixel data information. Among them, 2a is the total number of pixels in one row of the sensor pixel array, and 2b is the total number of pixels in one column of the pixel array.

[0043] The pixel memory array can be designed in three different ways: 1) Serial reading generates one coordinate pixel unit; 2) Parallel reading generates a row of coordinate pixel units; 3) Simultaneously reading all pixels generates all coordinate pixel units.

[0044] Figure 3 The working process embodiment of the chip applying the present invention is shown as follows.

[0045] After the 3D image sensor chip emits a line of light from the light source, the working process of imaging, pixel coordinate acquisition, and generating the 3D coordinates formed by the reflection and exposure of this line on the object to be measured is as follows:

[0046] 301. Control the active light source to emit light and project a line of light onto the object to be measured;

[0047] 302. The 3D image sensor controls pixel exposure;

[0048] 303. Select rows and columns to control and lock the pixels to be read;

[0049] 304. ADC analog-to-digital conversion of the pixels to be read;

[0050] 305. The pixel memory array sends the pixel data and pixel coordinates to the 3D point cloud generation unit;

[0051] 306. The 3D coordinate generation unit calculates the 3D point cloud coordinates X, Y, and Z using the pixel coordinates, the center point of the imaging lens of the image sensor, and the distance parameter between the sensor and the active light source.

[0052] After the above process completes the generation of the 3D coordinates of the pixel acquisition under a single light ray projected onto the object to be measured, the object to be measured is moved for imaging, and the above process is repeated to complete the generation of the 3D coordinates of the entire object.

[0053] Figure 4 The following shows the first embodiment of the application of the present invention

[0054] This figure is an embodiment of a 3D CMOS image sensor. Among them, the sensor pixel array is implemented by a CMOS sensor. Its advantages are high integration, small area, and low cost. The working principle is the same as Figure 1 described.

[0055] Figure 5 The following shows the second embodiment of the application of the present invention

[0056] This figure is an embodiment of a 3D CCD image sensor. Among them, the sensor pixel array is implemented by a CCD sensor. Its advantages are that the sensor is highly sensitive to light under low light conditions and has strong anti-interference ability. The working principle is the same as Figure 1 described.

[0057] Figure 6 The following shows the third embodiment of the application of the present invention

[0058] This figure is an embodiment of a 3D area array camera. System composition: laser emitter, laser galvanometer, 3D image sensor chip, 3D point cloud coordinate generation unit, optical lens mechanism. Its working principle is as follows:

[0059] 601. The laser emitter emits a light ray to the laser galvanometer, and the laser galvanometer projects the light ray onto the object to be measured;

[0060] 602. The 3D image sensor chip controls the exposure of the sensor pixel array;

[0061] 603. The pixel memory array reads the pixel exposure information and coordinates through row selection and column selection;

[0062] 604. The 3D point cloud generation unit calculates the 3D coordinates of the object to be measured corresponding to the exposed pixels of the laser beam projected by the 601 laser;

[0063] 605. The laser galvanometer rotates to the next angle and repeats 601 until all pixel columns are projected according to the resolution of the 3D image sensor of the area array camera.

[0064] Figure 7The following shows the fourth embodiment of the application of the present invention

[0065] This figure shows an embodiment of a 3D line-scanning camera. The system consists of a laser emitter, a 3D image sensor chip, a 3D point cloud coordinate generation unit, and an optical lens mechanism. The working principle is as follows:

[0066] 701. The laser emitter emits a ray of light to the laser galvanometer, and the laser galvanometer projects the light ray onto the object to be measured;

[0067] 702. The 3D image sensor chip controls the exposure of the sensor pixel array;

[0068] 703. The pixel memory array reads the pixel exposure information and coordinates through row selection and column selection;

[0069] 704. The 3D point cloud generation unit calculates the 3D coordinates of the object to be measured corresponding to the exposure pixels of the projection line of the 701 laser.

Claims

1. A device for implementing a 3D point cloud coordinate calculation system on an image sensor chip, characterized in that, the device is composed of a 3D image sensor chip, a 3D point cloud generation unit, a sensor array, an active light source and an optical lens mechanism. A pixel memory array, a row recorder and a column recorder are designed on the 3D image sensor chip. When the active light source emits light to the object to be measured, the row selection and column selection are controlled to read the exposure information of each pixel of the image sensor while recording the valid row selection and column selection addresses of the pixel, and then a coordinate pixel unit is generated, that is, pixel position coordinate information; The 3D point cloud generation unit calculates the 3D point cloud coordinates of the object to be measured according to the obtained coordinate pixel unit and the pre-calibrated center point parameters of the light source and the sensor lens; The device further includes the following steps: a. Control the active light source to emit light and project a ray of light onto the object to be measured; b. The 3D image sensor controls pixel exposure; c. Row selection and column selection controls lock the pixel to be read; d. Perform ADC analog-to-digital conversion on the read pixel; e. The pixel memory array sends the coordinate pixel unit formed by pixel data and pixel coordinates to the 3D point cloud generation unit; f. The 3D coordinate generation unit calculates the 3D point cloud coordinates X, Y, Z by using the pixel coordinates, the center point of the imaging lens of the image sensor, and the distance parameters between the sensor and the active light source; g. The object to be measured moves for imaging, and steps a to g are repeated until the generation of the 3D point cloud coordinates of the object to be measured is completed.

2. The device according to claim 1, characterized in that, The pixel memory array of the 3D image sensor chip integrates the pixel data information read from the DAC analog-to-digital conversion unit with the row address and column address information of the selected pixel to be read into a coordinate pixel unit, and sends the data of the coordinate pixel unit to the 3D point cloud coordinate generation unit. The pixel memory array filters and judges the read pixel information, eliminates the pixel data that fails to expose and image, and only sends the pixel information and coordinates that can expose and image to the 3D point cloud generation unit.

3. The device according to claim 1 or claim 2, characterized in that, The pixel memory array includes: 1) Reading one pixel serially to generate one coordinate pixel unit; 2) Reading a row of pixels in parallel to generate a row of coordinate pixel units; 3) Reading all pixels simultaneously to generate a global coordinate pixel unit.

4. The device according to claim 1, characterized in that, The active light source and the object to be measured move for imaging. The light source at least includes: 1) A line laser emitter light source; 2) An LED light source; The methods for the object to be measured to move for imaging include: 1) The object to be measured moves autonomously; 2) The 3D point cloud coordinate generation device moves autonomously; 3) Using a laser galvanometer to move the light emitted by the line laser.

5. The device according to claim 1, characterized in that, The method for the 3D point cloud generation unit to calculate the 3D point cloud coordinates of the object to be measured includes the following steps: a. Multiply the actual size of each pixel determined according to the integrated circuit production node process by the row and column address information where the pixel is located to calculate the actual coordinate value of the pixel; b. Establish an equation between the actual coordinate values and the object to be measured by using the principle of small hole imaging at the image center point of the optical lens; c. Calculate the 3D coordinates of the imaging pixels corresponding to the object to be measured by using the known distance data between the 3D image sensor and the active light source and the plane equation formed by the intersection of the light source ray and the object to be measured in space.

6. The device according to claim 1, characterized in that the sensor pixel array includes a physical sensor and a light filtering material, and the light filtering material includes: a light filtering material having the same wavelength as the light emitted by the active light source.

7. The device according to claim 6, wherein the physical sensor includes: a CMOS sensor, a CCD sensor.

8. The device according to claim 5, characterized in that the actual implementation method of the 3D point cloud generation unit includes: 1) Field Programmable Gate Array (FPGA); 2) Computing Unit (CPU) + Graphics Processing Unit (GPU); 3) System on Chip (SoC); 4) Integrated in the 3D image sensor chip and using the same integrated circuit process as the 3D image sensor.

9. The device according to claim 1 or claim 4, characterized in that the achievable device configurations include: 1) 3D area array camera; 2) 3D line scan camera; 3) 3D lidar device.

Images