Sensing, storing and computing integrated binocular machine vision 3D point cloud computing method and system

Abstract

The invention discloses a sensing, storing and computing integrated binocular machine vision 3D point cloud computing method and system, the system forms a binocular 3D vision sensor by a master sensing, storing and computing integrated chip and a slave sensing, storing and computing integrated chip based on a parallax computing strategy, and the slave sensing, storing and computing integrated chip comprises an image sensor, a pixel memory and a preprocessing computing circuit. The preprocessing circuit determines the pixels of which the photosensitive gray levels of the current pixel array frame and the previous frame change, the transmission module only transmits the changed pixels to the main sensing, storage and calculation integrated chip, the main sensing, storage and calculation integrated chip comprises an image sensor, a main-slave pixel memory and a parallax 3D point cloud calculation circuit, and the 3D point cloud reconstruction method has the advantage that the 3D point cloud reconstruction frame rate can be improved.

Description

Technical Field

[0001] This patent belongs to the fields of machine vision, artificial intelligence, and autonomous driving. Background Art

[0002] With the rapid application of CMOS optoelectronic sensor cameras in the field of autonomous driving, using 3D machine vision of binocular cameras to achieve distance measurement and object recognition in navigation will be more advantageous than lidar LiDAR. However, due to the limitations of in-vehicle CPU and GPU computing power, the resolution of current 3D machine vision solutions for in-vehicle CMOS image sensors with a frame rate of 30 frames per second is mostly VGA or 720p. Although the current resolution of CMOS image sensors has exceeded that of lidar, due to the low resolution, it is still impossible to identify objects and road signs at close and medium distances in real time. Therefore, an in-vehicle 2D camera with more than 5 million pixels must be used to identify objects on the road. Since the current 3D machine vision solution is based on dual 2D cameras connected to the CPU and GPU via USB3.0 or MIPI, when the resolution of the 2D camera reaches 5 million pixels, the current 3D machine vision solution can only achieve a 3D point cloud reconstruction frame rate of 3 to 5 frames per second. For example, GANet15 takes 1.8s to process a pair of images with a resolution of 1242×375 (TESLA P40 GPU); PSMNet takes 0.41s (TitanXp GPU); AANet takes 62ms (NVIDIA V100 GPU); and the real-time networks BGNet and BGNet+ also take 25.4ms and 32.3ms respectively (GTX 2080Ti). The price of high-end graphics cards, such as the NVIDIA RTX 3090Ti GPU, exceeds 10,000 yuan, and the current technology and products cannot meet the requirements of autonomous driving. Summary of the Invention

[0003] To solve the above problems, the present invention discloses a method and system for computing 3D point clouds in a sense-compute-integrated binocular machine vision. The system consists of a master and a slave sense-compute-integrated chip to form a binocular 3D vision sensor. Based on the disparity calculation strategy, the slave sense-compute-integrated chip includes an image sensor, a pixel memory, and a preprocessing calculation circuit. The preprocessing circuit determines the pixels with changed photosensitive grayscale between the current pixel array frame and the previous frame. The transmission module only transmits the changed pixels to the master sense-compute-integrated chip. The master sense-compute-integrated chip includes an image sensor, a master-slave pixel memory, and a disparity 3D point cloud computing circuit. The advantage of the present invention is that the calculation of the disparity map and the generation and reconstruction of the 3D point cloud are completed on two master-slave CMOS image sensor chips. A pixel difference comparison circuit is designed in the slave sense-compute-integrated chip, and a master-slave pixel array memory is designed in the master sense-compute-integrated chip, which can quickly generate all the pixels of the current frame for computing the disparity, reduce the data transmission volume between the two image sensor chips, improve the reconstruction frame rate of the 3D point cloud, and realize 3D point cloud computing on the chip. The overall cost of the vehicle-mounted 3D machine vision system can be reduced to less than $100. The present invention can be applied to the navigation and object recognition scenarios of autonomous driving and service robots.

[0004] To achieve the above object, the technical solution of the present invention is realized as follows.

[0005] A method and system for computing 3D point clouds in a sense-compute-integrated binocular machine vision. The system consists of a master and a slave sense-compute-integrated chip and related optical lenses to form a binocular 3D vision sensor, or a left-right binocular 3D vision sensor. The master sense-compute-integrated chip is responsible for completing the disparity and 3D point cloud computing of the current master-slave two-frame pixels. The slave sense-compute-integrated chip preprocesses the current pixels to be matched on the chip. The preprocessed pixels are transmitted through a high-speed data line connecting the master and slave sense-compute-integrated chips as a carrier. The length of the data line is at least the baseline length of the binocular 3D vision sensor. The slave sense-compute-integrated chip preprocesses the pixel array to be transmitted. Preferably, the preprocessing includes determining the pixels that change between the current pixel frame and the previous pixel frame.

[0006] The disparity calculation algorithm is as follows: Given a pixel point (x, y) in the main pixel array, to match and determine its corresponding pixel point (x R , y R ) in the slave pixel array, d = x - x R (y = y R ). The relationship between the depth coordinate Z and the disparity d is Z = BF / d, where B is the baseline length of the binocular vision system and F is the equivalent focal length. In the autonomous driving scenario, the maximum distance for forward vision ranging needs to reach 250 meters, and the baseline length of the binocular 3D vision system is about 1 meter. The key of the present invention lies in designing a preprocessing circuit in the slave sense-compute-integrated chip.

[0007] The sensor-in-memory integrated chip includes: 1) an image sensor slave pixel array; 2) a read pixel window; 3) a difference calculation circuit; 4) a memory for the previous frame of the slave pixel array; 5) a transmission module; The working process of the chip is as follows:

[0008] Step 1: The image sensor slave pixel array senses the object to be measured, forming the current slave pixel array;

[0009] Step 2: The read pixel window reads the current slave pixel array according to the disparity calculation strategy;

[0010] Step 3: The difference calculation circuit compares and calculates the pixels in the read pixel window with the pixels in the corresponding pixel window of the previous frame to obtain a changed pixel matrix;

[0011] Step 4: The transmission module transmits the changed pixels of the slave pixel array to the main sensor-in-memory integrated chip, and at the same time updates the corresponding pixels in the memory of the previous frame of the slave pixel array with the changed pixels;

[0012] Step 5: Go to Step 2 until all pixels in the current slave pixel array are processed;

[0013] Step 6: Go to Step 1 to perform the photosensitive processing work of the next frame.

[0014] Disparity calculation is to calculate the matching cost at a point C(x, y, d) in the 3D cost space C. C represents the matching cost at pixel (x, y) with a disparity of d. In the present invention, the strategy of reading pixels from the image sensor array is adapted to the fast calculation rule of the matching cost as much as possible.

[0015] Preferably, the row selection and column selection for the image sensor reading read pixels according to the requirements of the read pixel window. The read pixel window is the pixel area required to calculate the initial sparse disparity value for disparity calculation. Taking the Semi-Global Matching (SGM) algorithm and a 5 million pixel resolution

[0016] (2560x1920) 8-bit grayscale CMOS image sensor as an example, the matching cost matrix is cost[2559:0][1919:0]. The setting of the read pixel sensor can be 1960 bytes per pixel column or divide 1960 bytes into several segments.

[0017] Preferably, based on the neural network matching cost method, the read pixel window is set as an image block, 9x9 pixels.

[0018] Preferably, the difference calculation circuit is composed of multiple comparison calculation units. The number of calculation units is the same as the number of pixels read each time, so that the read pixel window and multiple pixels of the previous frame of the slave pixel array can be compared and calculated in parallel. The comparison calculation includes subtraction operations.

[0019] For the changing images collected at a high frame rate of 30 frames per second, the number of pixel changes between two adjacent frames is usually limited. In the present invention, the pixels with changes in the current frame are identified from the sense-compute-integrated chip and sent to the main sense-compute-integrated chip, thereby reducing the amount of data transmission and improving efficiency. Preferably, a method for calculating the change of the same pixel after two consecutive photosensitivities is to subtract the two photosensitive gray values and compare them with a set threshold. If it is greater than the threshold, the pixel is considered to have changed; if it is less than or equal to the threshold, the pixel is considered not to have changed.

[0020] Preferably, the changing pixel matrix of the sense-compute-integrated chip is a binary 01 matrix, and each bit in the 01 matrix corresponds to the position of the corresponding pixel in the pixel array of the image sensor. Among them, 0 indicates that the pixel has not changed, and 1 indicates that the pixel has changed. The 01 changing pixel matrix will be sent to the main sense-compute-integrated chip, and the receiving module determines the position of the received pixel in the previous frame from the pixel array memory.

[0021] The main sense-compute-integrated chip includes: 1) the main pixel array of the image sensor; 2) the read pixel window; 3) the current frame memory of the main pixel array; 4) the current frame memory of the slave pixel array; 5) the receiving module; 6) the disparity calculation circuit; 7) the 3D point cloud calculation circuit. The working process of the chip is as follows:

[0022] Step 1: The main pixel array of the image sensor photosensitizes the object to be measured to form the current main pixel array;

[0023] Step 2: The receiving module receives the changing pixels from the slave pixel array in units of the read pixel window, and updates the current frame memory of the slave pixel array according to the identification of the 01 matrix to form a slave pixel array corresponding to the current main pixel array;

[0024] Step 3: The disparity calculation circuit calculates the disparity for the slave pixel array corresponding to the current main pixel array;

[0025] Step 4: The 3D point cloud calculation circuit converts the disparity into a 3D point cloud;

[0026] Step 5: Go to Step 2 until all pixels of the current main and slave pixel arrays are processed;

[0027] Step 6: Go to Step 1 to perform the photosensitization processing work for the next frame.

[0028] Preferably, the read pixel window can be implemented by a register bank or SRAM.

[0029] Preferably, the current frame and the previous frame memories can be implemented by SRAM. Calculated with 5 million pixels and 8-bit gray scale, the memory requires 5 MByte.

[0030] Preferably, the parallax calculation circuit and the parallax calculation strategy are implemented based on the Semi-Global Matching (SGM) algorithm. Description of the Drawings

[0031] Figure 1 Schematic diagram of the binocular master-slave sense-memory-computation integrated chip and system architecture applying this patent

[0032] Figure 2 For the second embodiment of applying this patent, a vehicle-mounted autonomous driving 3D machine vision system Detailed Description of the Invention

[0033] Appendix Figure 1 Schematic diagram of the binocular master-slave sense-memory-computation integrated chip and system architecture of this patent

[0034] 101. Master sense-memory-computation integrated chip, comprising: an image sensor, a memory, and a parallax 3D point cloud calculation circuit, responsible for calculating parallax and 3D point cloud;

[0035] 1011. Image sensor, main pixel array, which is a CMOS image sensor, including row-column control for pixel reading and ADC conversion circuit;

[0036] 1012. Read pixel window, a high-speed buffer (buffer) for optimally reading the main pixel array determined according to the parallax calculation algorithm, which is an input parameter of the parallax calculation circuit;

[0037] 1013. Current frame memory of the main pixel array, storing the pixels of one frame of the main pixel array;

[0038] 1014. Current frame memory of the slave pixel array, storing the current slave pixel array;

[0039] 1015. Receiving module, receiving the changed pixels and 01 change matrix from the slave sense-memory-computation integrated chip, updating the current frame memory of the slave pixel array according to the marks of the 01 changed pixel matrix, and then generating a new current frame of the slave pixel array

[0040] 1016. Parallax calculation circuit, taking the read pixel window or the current frame memory of the main pixel array as the input, using the parallax algorithm to match the same pixels of the detected object in the slave pixel array, and calculating the parallax;

[0041] 1017. 3D point cloud calculation circuit, calculating the 3D point cloud from the parallax and the pixel matrix;

[0042] 102. Slave sense-memory-computation integrated chip, comprising: an image sensor, a memory, and a preprocessing calculation circuit;

[0043] 1021. Image sensor, slave pixel array, which is a CMOS image sensor, including row-column control for pixel reading and ADC conversion circuit;

[0044] 1022. Read the pixel window. The high-speed buffer buffer of the main pixel array read optimally determined according to the disparity calculation algorithm is an input parameter of the difference calculation circuit;

[0045] 1023. The difference calculation circuit calculates the gray-scale difference by subtracting the gray-scale values of the pixels in the current read pixel window from the corresponding row and column pixels in the previous frame memory of the pixel array.

[0046] 1024. The sending module uses a threshold for the gray-scale difference of the pixels to determine the change of the pixels and generates a 01 change matrix.

[0047] 1025. After completing the pixel difference calculation, store the current pixel in the memory from the previous frame memory of the pixel array.

[0048] Implementation of the first embodiment of this patent based on the SGM algorithm chip

[0049] In this embodiment, a binocular 3D machine vision sensor with 5 million pixels is used as the target. The system can achieve 3D point cloud, ranging, and 3D object registration and recognition at 30 frames per second. At the same time, the main sense-in-memory computing chip is used to complete the ISP processing to form a 2D RGB image, providing data for the in-vehicle system to achieve high-precision 2D image recognition.

[0050] Image sensor, both the main and slave pixel arrays are 2560x1920, and an 8-bit ADC is used as the pixel gray scale.

[0051] According to the calculation strategy of the SGM matching cost space, the parallel design of the chip has 8 Pipelines. The read pixel window is 1920 / 8 = 240 bytes. The difference calculation circuit of the slave sense-in-memory computing chip is 240 parallel subtractors. The transmission protocol between the main and slave sense-in-memory computing chips is LVDS (Low-Voltage Differential Signaling). 8-channel LVDS is used. The baseline of the camera main and slave sensors is 1 meter. The total transmission rate is 8x200Mbps = 200MBps. Theoretically, the full-frame transmission rate of 5 million pixels can reach 40 frames per second. Using the effective changed pixels after the difference calculation of this patent at 50%, the transmission rate can reach 80 frames per second.

[0052] Figure 2 For the second embodiment of this patent, the in-vehicle autonomous driving 3D machine vision system

[0053] 201. The main camera optical lens based on the main sense-in-memory computing chip, with a focal length of F;

[0054] 202. The main sense-in-memory computing chip, calculating the disparity and 3D point cloud;

[0055] 203. ISP image signal processor, which processes the pixel array of the main sense-in-memory computing integrated chip to generate RGB 2D image data;

[0056] 204. In-vehicle main computing platform, whose hardware consists of a CPU, a GPU, and an application-specific integrated circuit, and mainly includes: 1) 2D object recognition, which recognizes the RGB image data generated by the ISP, including road signs, the license plate of the vehicle in front, road pedestrians, etc.; 2) 3D object recognition, which recognizes the objects on the road and the distance between the objects and the vehicle; the main computing platform sends the results of 2D object recognition and 3D object recognition to the vehicle driving control system to achieve autonomous driving.

[0057] 205. Main camera optical lens based on the sense-in-memory computing integrated chip, with a focal length F consistent with 201;

[0058] 206. Sense-in-memory computing integrated chip, which preprocesses the currently collected image and sends the changed effective pixels to the main sense-in-memory computing integrated chip;

[0059] 207. Binocular 3D machine vision system baseline B and transmission line, and the transmission line adopts the LVDS protocol.

[0060] In this embodiment, F = 200 mm, the viewing angle is 12 degrees, B = 1800 mm, the pixel length patch of the image sensor is 3 um, the row length of 5 million pixels is 7 mm, the parallax d for the maximum ranging is 7 mm / 2, and the maximum measurable distance in this embodiment is 102 meters.

Claims

1. A method and system for calculating 3D point cloud in a sense-storage-computation integrated binocular machine vision, characterized in that A binocular 3D vision sensor system consists of a master and a slave in-memory computing chip. Based on the disparity calculation strategy, the slave in-memory computing chip preprocesses the pixels, calculates the difference change between the current pixel and the pixel in the previous frame, and transmits the changed pixels to the master in-memory computing chip through the transmission module. The master in-memory computing chip quickly generates the current slave pixel array and calculates the disparity and 3D point cloud for the master and slave pixel arrays.

2. The system according to claim 1, characterized in that The slave in-memory computing chip at least includes: 1) an image sensor slave pixel array; 2) a pixel reading window; 3) a difference calculation circuit; 4) a memory for the previous frame of the slave pixel array; 5) a transmission module. The working process of the chip is as follows: Step 1: The image sensor slave pixel array senses the object to be measured and forms the current slave pixel array. Step 2: The pixel reading window reads the current slave pixel array according to the disparity calculation strategy. Step 3: The difference calculation circuit compares and calculates the pixels in the pixel reading window with the corresponding pixels in the previous frame's pixel reading window to obtain the changed pixel matrix. Step 4: The transmission module transmits the changed pixels of the slave pixel array to the master in-memory computing chip, and at the same time updates the corresponding pixels in the memory for the previous frame of the changed pixels in the slave pixel array. Step 5: Go to Step 2 until all pixels in the current slave pixel array are processed. Step 6: Go to Step 1 to perform the light-sensing processing work for the next frame.

3. The system according to claim 1, characterized in that The master in-memory computing chip is characterized by at least including: 1) an image sensor master pixel array; 2) a pixel reading window; 3) a memory for the current frame of the master pixel array; 4) a memory for the current frame of the slave pixel array; 5) a receiving module; 6) a disparity calculation circuit; 7) a 3D point cloud calculation circuit. The working process of the chip is as follows: Step 1: The image sensor master pixel array senses the object to be measured and forms the current master pixel array. Step 2: The receiving module receives the changed pixels of the slave pixel array in units of the pixel reading window, updates the memory for the current frame of the slave pixel array, and forms a slave pixel array corresponding to the current master pixel array. Step 3: The disparity calculation circuit calculates the disparity for the slave pixel array corresponding to the current master pixel array. Step 4: The 3D point cloud calculation circuit converts the disparity into a 3D point cloud. Step 5: Go to Step 2 until all pixels in the current master and slave pixel arrays are processed. Step 6: Go to Step 1 to perform the light-sensing processing work for the next frame.

4. The method according to claims 1 and 2, characterized in that The preprocessing of pixels by the slave in-memory computing chip refers to the comparison result between the subtraction of the grayscale values of the current pixel and the pixel in the previous frame at the same row and same column position and a preset threshold, forming a 01 changed pixel matrix.

5. The system according to claims 1 and 2, characterized in that The content that the transmission module transmits the changed pixels to the master in-memory computing chip at least includes: 1) based on the changed pixel grayscale values in the pixel reading window; 2) the 01 changed pixel matrix.

6. The system according to claims 1 and 2, characterized in that The disparity calculation strategy and the pixel reading window refer to the method of determining the size of the pixel reading window based on the calculation unit for calculating the matching cost according to the disparity algorithm, and then determining the row selection and column selection pixel reading strategies of the image sensor to read pixels according to the requirements of the pixel reading window, including: 1) the pixel area required for calculating the initial sparse disparity value; 2) reading by column; 3) reading by image block based on a neural network.

7. The system according to claims 1, 2 and 3, characterized in that The working process of the master-slave pixel array for calculating parallax and the chip refers to the synchronous operation of the master and slave sensor-in-memory-and-computation integrated chips.

8. The system according to claim 1, characterized in that The binocular 3D vision sensor and the transmission of changing pixels to the master sensor-in-memory-and-computation integrated chip through the transmission module mean that it is transmitted through the high-speed data line connecting the master and slave sensor-in-memory-and-computation integrated chips as a carrier, and the length of the data line is at least the baseline length of the binocular 3D vision sensor.

Images