Binocular 3D machine vision device without active light source

Abstract

The invention discloses a 3D machine vision device without an active light source. A system is composed of a left camera, a right camera, an image large model reasoning unit and a 3D machine vision unit. The implementation method of the 3D machine vision device is as follows: 1) the left camera images a measured object, and the object recognition unit recognizes the measured object; (2) imaging the measured object by the right camera, identifying the measured object by an image large model reasoning unit, and determining that the measured object and the measured object of the left camera are the same object; 3) the left camera determines a unique feature pixel in the image of the measured object; (4) the right camera searches for pixels matched with the unique feature pixels of the left camera in the image near the base line; 5) the 3D machine vision unit calculates the parallax formed by the unique feature pixel on the left camera and the right camera, and further calculates the distance value and the 3D point cloud of the measured object; and 6) completing 3D reconstruction of the detected object based on the 3D model of the large model detected object by using the 3D coordinate of the unique feature pixel and the 2D shape of the detected object. The method has the advantages that an active light source is not needed, the detected object in the complex environment is recognized on the left camera and the right camera through image large model reasoning, and the distance of the detected object can be measured in the outdoor environment with strong ambient light to calculate the 3D point cloud.

Description

Technical Field

[0001] This patent belongs to the fields of artificial intelligence, embodied intelligence, 3D machine vision, and large-scale model reasoning technology. Background Technology

[0002] Active-light 3D cameras are greatly affected by ambient light, making them unsuitable for complex outdoor scenarios such as humanoid robots, aerial surveying and terrain 3D modeling, drones, and autonomous driving. Traditional 3D cameras without active light sources typically rely on global, semi-global, or machine learning algorithms like SGM to search for similar pixels and calculate parallax. The main problems with this approach are: 1) High computational cost: for a 720p resolution binocular camera, calculating the depth map using a high-performance GPU requires a frame rate of 20 frames per second; 2) Low accuracy: in autonomous driving scenarios, the low resolution of 2D images results in distance measurement accuracy at the 10-centimeter level. In aerial surveying scenarios, when the distance to the detected object is greater than 100 meters from the ground, the distance measurement accuracy is only above 10 meters, failing to meet the required precision.

[0003] In scenarios such as outdoor fruit picking, tea picking, equipment inspection, and high-speed navigation, it is only necessary to identify and measure the distance of various objects being detected in the scene. Summary of the Invention

[0004] To address the aforementioned issues, this invention discloses a 3D machine vision device without an active light source, which calculates the distance information of the detected object. The system comprises a left camera, a right camera, an image large-scale model inference unit, and a 3D machine vision unit. The implementation method of the 3D machine vision device is as follows: 1) The left camera images the detected object, and the object recognition unit identifies the detected object; 2) The right camera images the detected object, and the image large-scale model inference unit identifies the detected object and determines that the detected object and the detected object in the left camera are the same object; 3) The left camera identifies a unique feature pixel in the image of the detected object; 4) The right camera searches for a pixel in the image near the baseline that matches the unique feature pixel of the left camera; 5) The 3D machine vision unit calculates the parallax formed by the unique feature pixel in the left and right cameras, and then calculates the distance value and 3D point cloud of the detected object; 6) Using the 3D coordinates of the unique feature pixel and the 2D shape of the detected object, the 3D reconstruction of the detected object is completed through the 3D model of the detected object in the large-scale model. The advantage of this invention lies in the fact that it does not require an active light source. It uses the object being detected as the target, such as an apple in an agricultural apple-picking scenario. It utilizes a large image model for inference to identify the object in complex environments via left and right cameras. In outdoor environments with strong ambient light, it can measure the distance to the detected object and calculate 3D point clouds. This invention's active light-free 3D machine vision device provides visual perception capabilities for embodied robots.

[0005] To achieve the above objectives, the technical implementation of the present invention is as follows:

[0006] A fixed distance *b* is set between the left and right cameras, and the focal length of the cameras is *f*. The left and right cameras undergo pre-processed 3D stereo correction of their baselines. The left and right cameras are connected to an image large-scale model inference unit. This unit receives two 2D images from the left and right cameras to identify the same object. To adapt to various environments, preferably, the left and right cameras are grayscale cameras, RGB color cameras, or infrared cameras. Preferably, the large model library of the image large-scale model inference unit is obtained by training on all detected objects in the robot's working scene.

[0007] The 3D machine vision unit calculates the z-axis distance and 3D point cloud of the detected object based on the image results of the same object in the left and right cameras, provided by the image large model inference unit. The left camera then determines the unique feature pixel coordinates (x, y, z) in the image of the detected object. l, y l Ideally, to reduce the computational cost of the global search, a Region of Interest (ROI) with large pixel value variations is selected to determine the coordinates of unique feature pixels. The right camera, in its coordinate system, is positioned at (x... l, y l Find the unique feature pixel of the left camera within the ROI region centered at (x, y). The resulting pixel is located at the coordinates (x, y) of the right camera. r ,y r The pixel used to calculate disparity is d = x. The 3D machine vision unit calculates the disparity d = x. r -x l The depth coordinates z = b*f / d of the unique feature pixel of the detected object are obtained.

[0008] Preferably, based on the 3D coordinates of the unique feature pixels of the detected object, the 2D graphic of the detected object, and the 3D model of the detected object recorded in the large model, the current detected object is reconstructed in 3D to provide the robot with an accurate trajectory for manipulating the detected object. Attached Figure Description

[0009] Figure 1 A schematic diagram illustrating the principle of applying this invention.

[0010] Figure 2 To apply the first embodiment of the present invention

[0011] Figure 3 To apply the second embodiment of the present invention Detailed Implementation

[0012] Figure 1 A schematic diagram illustrating the principle of applying this invention.

[0013] 101. The left and right cameras of the binoculars image the scene of the object being measured. The distance between the center point of the left camera and the center point of the right camera is b.

[0014] The 102 large model inference units identify the same detected object in the left and right cameras. According to the target to be detected, the detected object is identified in the image of the left camera, and the same detected object is matched in the image of the right camera using the detected object identified by the left camera as the target.

[0015] 103. Determine the unique feature pixel (x) in the left camera image. l ,y l Select a Region of Interest (ROI) with large pixel value variations, identify a pixel with a unique pixel value that differs significantly from other pixels in the ROI;

[0016] 104 Match unique feature pixels (x) within the right camera baseline ROI window. r ,y r );

[0017] 105. Calculate the disparity d = x r -x l And z = b*f / d, generate the coordinates (x) of the unique feature pixel. l ,y l ,z);

[0018] 106. Utilize a large model library of 3D models to complete the 3D reconstruction of the detected object, using the coordinates (x, y) of unique feature pixels. l ,y l Using z as the base point, the detected object is reconstructed in 3D based on the 3D model matched in the large model library.

[0019] Figure 2 To apply the first embodiment of the present invention

[0020] This embodiment illustrates the implementation architecture of the 3D machine vision device of the present invention, used in agricultural fruit picking scenarios. It has a maximum working field of view of 2000mm x 3000mm and a measurement accuracy of 1mm, and consists of the following four parts:

[0021] 201. Left camera, RGB color camera, 12-megapixel resolution, the camera and the computer of the image large model inference unit are connected via a high-speed USB interface;

[0022] 202. Right camera, RGB color camera, 12 megapixel resolution, the camera and the computer of the image large model inference unit are connected by a high-speed USB interface, and the center distance between the left and right cameras is b = 300mm;

[0023] 203. Image large model inference unit: The computer system is equipped with CPU and GPU to perform inference and recognition calculations on all different detected objects in the scene locally, and detect the target detected object (fruit). The large model library includes models such as fruit trees, fruits, leaves, tree trunks, humans, and birds.

[0024] The 204.3D machine vision unit is software running on a computer system. It selects a unique feature pixel from the 2D image of the object being detected in the left camera, and searches for a matching pixel within the baseline ROI window in the 2D image of the right camera. It then calculates the parallax, performs 3D reconstruction of the object, and provides the 3D coordinates of the object to the robot's trajectory generation system.

[0025] 205. Cloud-based large image model inference unit: When the model library and computing power of the local large image model inference unit 203 cannot meet the requirements, the cloud-based large image model library is called and the inference task is assigned to cloud computing.

[0026] Figure 3 For applying the second embodiment of the present invention

[0027] This embodiment verifies the application of the present invention's active light source-free 3D machine vision device with an apple as the target object to be detected.

[0028] Imaging and recognition of the apple by the 301 left camera;

[0029] Imaging and recognition of the same apple by the 302 right camera;

[0030] 303 is the unique feature pixel of the left camera;

[0031] Pixels that match the unique feature pixels of the right camera image and the left camera image (304) are used to detect feature points of the apple images of the left and right cameras respectively using the SURF (Speeded UpRobust Features) feature detector. Then, the disparity is calculated by matching the detected feature points.

[0032] In this embodiment, the average accuracy of distance measurement for the target object, an apple, is 0.5 mm, which meets the accuracy requirements for the robot to pick apples.

Claims

1. A 3D machine vision device without active light source, comprising: a left camera, a right camera, an image large model inference unit, and a 3D machine vision unit; the implementation method of the 3D machine vision device of the present invention is as follows: 1) The left camera images the object under test, and the object recognition unit recognizes the object under test; 2) The right camera images the object under test, the image large model inference unit recognizes the object under test, and determines that the object under test and the object under test of the left camera are the same object; 3) The left camera determines a unique feature pixel in the image of the object under test; 4) The right camera searches for a pixel in the image near the baseline that matches the unique feature pixel of the left camera; 5) The 3D machine vision unit calculates the disparity formed by the unique feature pixel in the left and right cameras, and then calculates the distance value and 3D point cloud of the object under test; 6) Using the 3D coordinates of the unique feature pixel and the 2D shape of the object under test, based on the 3D model of the object under test in the large model, the 3D reconstruction of the object under test is completed.

2. The apparatus according to claim 1, characterized in that, The image large model inference unit for recognizing the tested object is an image large model inference system, which includes at least: 1) machine learning to train image models of multiple detected objects and recognize them by multi-model inference; 2) Establish a template model for the object to be detected and identify it by matching the template.

3. The apparatus and method according to claim 1, characterized in that, The left camera determines the unique feature pixel coordinates (x) in the image of the object being measured. l, y l The method should include at least: 1) grayscale values ​​as features; 2) RGB color values ​​as features.

4. The apparatus and method according to claim 1, characterized in that, The right camera searches for pixels in the image near the baseline that match the unique feature pixel of the left camera (x). r ,y r ) refers to the coordinate system of the right camera, with (x... l, y l Search within a ROI window area centered on the left camera for pixels that match the unique feature pixel of the left camera. r ,y r ).

5. The apparatus and method according to claim 1, characterized in that... The 3D machine vision unit calculates the parallax d = x formed by the unique feature pixels in the left and right cameras. r -x l The depth coordinates of the unique feature pixel are z = b * f / d.

6. The apparatus and method according to claims 1 and 5, characterized in that... The 3D machine vision unit calculates the parallax formed by the unique feature pixel in the left and right cameras. The device is designed with a fixed distance b between the left and right cameras and a focal length f between the left and right cameras. The left and right cameras are pre-stereocorrected to form the same baseline.

7. The apparatus and method according to claim 1, characterized in that, The left and right cameras are 2D vision sensors, and contain at least: 1) Grayscale camera; 2) RGB color camera; 3) Infrared camera.

8. The apparatus according to claim 1, characterized in that... The image large model inference unit consists of a library of detected object models, inference software, and a computer system. Its working process is as follows: Step 1. The computer system receives images from the left and right cameras; Step 2. The inference software calls the large model library to identify the objects to be detected in the image from the left camera; Step 3. The inference software calls the large model library to identify the detected objects in the image from the right camera and matches them with the detected objects from the left camera.

9. The apparatus according to claim 1, characterized in that... The system composition refers to the connection of the left and right cameras to the computer system of the image large model inference unit, and the connection of the 3D machine vision unit to the image large model inference unit. Its working process is as follows: Step 1. The left and right cameras acquire images of the detected object and transmit them to the image large model inference unit; Step 2. The image large model inference unit identifies the same detected object in both the left and right cameras; Step 3. The 3D machine vision unit calculates the parallax of the detected object between the left and right cameras, and then calculates the distance information of the detected object.

10. The apparatus according to claim 1, characterized in that... The 3D reconstruction of the detected object follows these steps: Step 1. Based on the 2D image of the currently detected object, perform 2D image matching on models in the large model library to obtain the best matching 3D model; Step 2. Determine the position of the unique feature pixel of the detected object in the best matching 3D model, and perform 3D reconstruction pixel by pixel.

Images