High-precision three-dimensional point cloud measurement system and measurement method

CN118936305APending Publication Date: 2024-11-12CHANGCHUN INST OF OPTICS FINE MECHANICS & PHYSICS CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410986406.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing binocular vision measurement system has a single target recognition in complex lighting environments, low measurement accuracy and insufficient anti-interference ability.

Method used

Using a high-precision three-dimensional point cloud measurement system combining binocular and speckle structured light, speckle is projected to the target through the speckle projection unit in the front-end image detection device. Two near-infrared cameras perform binocular measurements, and the observation camera obtains RGB images , combined with SGBM algorithm, Mask R-CNN network and PnP/ICP algorithm, it realizes the generation and pose calculation of point cloud graphs.

Benefits of technology

It realizes three-dimensional point cloud measurement with multi-object recognition, high accuracy, stable and fast and strong anti-interference ability in complex lighting environments, and is suitable for robot grabbing and operation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118936305A_ABST
    Figure CN118936305A_ABST
Patent Text Reader

Abstract

The invention relates to a high-precision three-dimensional point cloud measurement system, relates to the technical field of computer vision technology application, and solves the problems that existing vision measurement equipment is relatively weak in complex illumination resistance, single in recognized target and relatively low in measurement precision. The measurement system comprises a front-end image detection device, a comprehensive controller and an upper computer which are connected in sequence, the front-end image detection device comprises a speckle projection unit, two near-infrared cameras and an observation camera. According to the high-precision three-dimensional point cloud measurement system, the three-dimensional point cloud is generated in a binocular and speckle structured light mode, the speckle projection unit projects speckles to a target area, after the binocular camera performs imaging, imaging data is processed by the comprehensive controller to generate point cloud data, and recognition and pose calculation of a target tool are completed through an algorithm. The three-dimensional point cloud measurement system has the advantages of multiple targets, high precision, stability, rapidness and strong anti-interference capability, and can be applied to environments with complex illumination.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the application field of computer vision technology, and in particular to a high-precision three-dimensional point cloud measurement system and a measurement method. Background Art

[0002] With the increasing requirements for automation in industrial production, computer vision technology has been widely used in the robotics industry. Robots equipped with visual assistance systems can more accurately and quickly complete tasks such as target grasping, assembly, and handling. The visual assistance system identifies various target tools, calculates the position and posture of the target tools, and guides the robot to grasp the target tools and perform other operations. Usually, the visual assistance system takes two-dimensional images as the research object, and converts two-dimensional information into three-dimensional information through visual models to obtain the position and posture of the research object. The most commonly used methods of visual measurement technology include: monocular measurement technology, binocular measurement technology, and multi-eye measurement technology. Among them, binocular measurement is more often used in non-cooperative target measurement systems due to its simple structure, high accuracy, and low cost.

[0003] There are many excellent research and application results on binocular vision measurement technology at home and abroad. Osaka University in Japan used an adaptive binocular visual servo system to complete the visual tracking of unknown moving objects. This method uses the principle of binocular vision to calculate the Jacobian matrix of the moving target image to infer the movement direction of the target object, thereby realizing the motion prediction and tracking of the target object. The Massachusetts Institute of Technology in the United States has studied a method for integrating a sensor system and a binocular vision system. This technology can be used in intelligent transportation systems. This method first uses a radar sensor to roughly provide the depth information range of the target, and then uses the binocular vision system to accurately calculate the depth information of the target. However, the above binocular measurement systems all have the problems of weak ability to resist complex lighting, single recognition target, and low measurement accuracy. Summary of the invention

[0004] The present invention aims to solve the technical problems in the prior art and provide a high-precision three-dimensional point cloud measurement system and a measurement method.

[0005] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0006] A high-precision three-dimensional point cloud measurement system, comprising: a front-end image detection device, an integrated controller and a host computer connected in sequence;

[0007] Among them, the front-end image detection equipment includes:

[0008] A speckle projection unit is arranged in the middle of the front-end image detection device and is used to project speckles onto the target;

[0009] Two near-infrared cameras are respectively set at the two ends of the front-end image detection device to perform binocular measurement and structured light measurement of the target;

[0010] The observation camera is set in the middle of the front-end image detection device to realize imaging of the target within the visible light range;

[0011] The front-end image detection equipment is used to project speckles onto the target, collect real-time images on site, receive and execute instructions sent by the host computer, and transmit image data to the integrated controller;

[0012] The integrated controller is used to receive images from the front-end image detection equipment, pre-process the point cloud generation algorithm, synchronize the camera, communicate, and control the speckle projection;

[0013] The host computer is used to control the entire system and display the system's working status, and transmit the target posture information to the front-end image detection device.

[0014] In the above technical solution, the front-end image detection device is installed on the robot.

[0015] The measurement method applicable to the above-mentioned high-precision three-dimensional point cloud measurement system includes the following steps:

[0016] Step 1: After the two near-infrared cameras collect image information, they use the SGBM algorithm for stereo matching to solve the disparity of each pixel and convert the disparity image into a point cloud image;

[0017] Step 2: Obtain RGB images through the observation camera and extract the target area image. Use the Mask R-CNN network to segment the color image. Determine the alignment relationship between each 3D point and each pixel through pixel alignment to complete the fusion of the image feature map and the point cloud feature map.

[0018] Step 3: Use the PnP algorithm and ICP algorithm to achieve coarse and fine registration of the point cloud map, complete the pose calculation of the target tool, and guide the robot to grasp and operate the tool.

[0019] In the above technical solution, step 1 specifically includes:

[0020] The Zhang Zhengyou calibration method is used to calibrate the internal and external parameters and distortion parameters of the near-infrared camera and observation camera to ensure the measurement accuracy of the measurement system;

[0021] The speckle projection unit projects speckles onto the target area, and two near-infrared cameras collect binocular images and transmit them to the integrated processor;

[0022] The integrated controller performs epipolar correction on the acquired binocular images, and then uses the SGBM algorithm for stereo matching to solve the disparity of each pixel and convert the disparity image into a point cloud image.

[0023] In the above technical solution, step 2 specifically includes:

[0024] The observation camera acquires the RGB image of the target and transmits it to the integrated controller, which extracts the target area image and uses the Mask R-CNN network to segment the color image.

[0025] The integrated controller determines the alignment relationship between each 3D point and each pixel by pixel alignment, and completes the fusion of the image feature map and the point cloud feature map.

[0026] The present invention has the following beneficial effects:

[0027] The high-precision 3D point cloud measurement system of the present invention realizes the generation of 3D point cloud by binocular and speckle structured light. The speckle projection unit projects speckle to the target area. After the binocular camera forms an image, the imaging data is processed by the integrated controller to generate point cloud data. The recognition and posture calculation of the target tool are completed through the algorithm. It is a multi-target, high-precision, stable, fast and strong anti-interference 3D point cloud measurement system, which can be used in environments with complex lighting. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0029] Figure 1 It is a structural schematic diagram of the high-precision three-dimensional point cloud measurement system of the present invention;

[0030] Figure 2 The software workflow diagram of the high-precision three-dimensional point cloud measurement system of the present invention;

[0031] Figure 3 This is a flow chart of the point cloud registration algorithm of the high-precision three-dimensional point cloud measurement system of the present invention.

[0032] Figure 4 This is a flow chart of the pose calculation of the high-precision three-dimensional point cloud measurement system of the present invention. DETAILED DESCRIPTION

[0033] The inventive concept of the present invention is: the present invention aims to solve the problems of weak ability of existing visual measurement equipment to resist complex lighting, single recognition target, and low measurement accuracy, and to solve the position and posture between the robot and the target tool. The high-precision three-dimensional point cloud measurement system of the present invention adopts binocular and speckle structured light to realize the generation of three-dimensional point cloud. The speckle projection unit projects speckle to the target area. After the binocular camera forms an image, the imaging data is processed by the integrated controller to generate point cloud data, and the recognition and posture calculation of the target tool are completed through the algorithm. The algorithm for completing the recognition and pose calculation of the target tool is divided into three steps: first, the binocular structured light camera collects image information and performs stereo matching through the SGBM algorithm to solve the disparity of each pixel point and convert the disparity image into a point cloud image; second, the RGB image is obtained through the observation camera and the target area image is extracted. The Mask R-CNN network is used to segment the color image, and the alignment relationship between each 3D point and each pixel is determined by pixel alignment to complete the fusion of the image feature map and the point cloud feature map; finally, the PnP algorithm and ICP algorithm are used to realize the coarse and fine registration of the point cloud map, complete the pose calculation of the target tool, and guide the robot to grasp and operate the tool.

[0034] The present invention is described in detail below with reference to the accompanying drawings.

[0035] The high-precision three-dimensional point cloud measurement system of the present invention realizes the output of three-dimensional point cloud by combining binocular and structured light.

[0036] like Figure 1 As shown ( Figure 1 Only some components are shown), the high-precision three-dimensional point cloud measurement system of the present invention includes three parts connected in sequence: a front-end image detection device (also called a measurement camera), an integrated controller and a host computer.

[0037] The measuring camera is used to project speckles onto the target, collect real-time images on site, receive and execute instructions sent by the host computer, and transmit image data to the integrated controller;

[0038] The integrated controller is used to receive images from the front-end image detection equipment, pre-process the point cloud generation algorithm, synchronize the camera, communicate, and control the speckle projection;

[0039] The host computer is used to control the entire system and display the system's working status, and transmit the target posture information to the robotic arm.

[0040] The measurement camera is installed on the robot and consists of 3 cameras and 1 speckle projection unit, including:

[0041] 2 near-infrared cameras (i.e. Figure 1Camera 1 and Camera 2 in the figure are located at the left and right ends of the measurement camera and serve as binocular cameras. The binocular images collected by the left and right binocular cameras are used for stereo matching and 3D reconstruction to realize binocular and structured light measurement.

[0042] The third camera (i.e. Figure 1 The middle camera 3) is located in the middle of the measuring camera. As an observation camera, it realizes imaging within the visible light range. The collected color images are used for target detection and segmentation. The left and right eye images, and the left eye and color images have been calibrated and registered by offline calibration software. The color image can establish a mapping relationship with the depth information.

[0043] The speckle projection unit is arranged in the middle of the measuring camera and is used to project speckles onto the target.

[0044] After the binocular camera forms an image, it transmits the image data to the integrated controller, which converts the image data into point cloud data and sends it to the host computer. The processing software inside the integrated controller performs tool target recognition and posture measurement, and performs three-dimensional reconstruction of the scene at the same time.

[0045] The measurement method of the high-precision three-dimensional point cloud measurement system applicable to the present invention specifically includes the following steps:

[0046] 1) The Zhang Zhengyou calibration method is used to calibrate the internal and external parameters and distortion parameters of the near-infrared camera and observation camera to ensure the measurement accuracy of the measurement system;

[0047] 2) The speckle projection unit projects speckles onto the target area, and two near-infrared cameras collect binocular images and transmit them to the integrated controller;

[0048] 3) The integrated controller performs epipolar correction on the acquired binocular images, and then uses the SGBM algorithm for stereo matching to solve the disparity of each pixel and convert the disparity image into a point cloud image;

[0049] 4) The observation camera acquires the RGB image of the target and transmits it to the integrated controller, which extracts the target area image and uses the Mask R-CNN network to segment the color image;

[0050] 5) The integrated controller determines the alignment relationship between each 3D point and each pixel by pixel alignment, and completes the fusion of the image feature map and the point cloud feature map;

[0051] 6) The PnP algorithm is used to achieve coarse alignment of the point cloud map, and then the ICP algorithm is used to achieve fine alignment of the point cloud map, complete the pose calculation of the target tool, and guide the robot to grasp and operate the tool.

[0052] The workflow of the processing software inside the integrated controller of the present invention is as follows: Figure 2 As shown in the figure, the binocular image generates a point cloud through stereo matching, and the color image collected synchronously is used to identify and segment the target to obtain a 2D target mask. The point cloud at the target can be segmented through the external parameter matrix. The segmented target point cloud is input into the pose measurement module, which calculates the conversion relationship between the real-time target and the model point cloud. Since the preset model point cloud coordinate system coincides with the camera coordinate system, the conversion relationship is the pose of the target in the camera coordinate system.

[0053] 1) Stereo matching

[0054] The classic SGBM algorithm is used for full-image stereo matching to solve the three-dimensional coordinates of each pixel in the image. This process is the process of converting the disparity image into a point cloud. When performing dense stereo matching under the binocular model, it is necessary to complete the extreme correction of the binocular image, which can reduce the disparity range that needs to be searched, thereby reducing the amount of calculation. The extreme correction work has two steps: image coplanarity and row alignment. Convert the rotation matrix R between the two infrared camera coordinate systems to the Rodriguez vector θ*u, where u is the unit rotation vector and θ is the rotation angle. Convert the vector Perform the inverse Rodriguez transformation to obtain the rotation matrix r, which satisfies R = r*r. Rotate the left infrared camera by r and the right infrared camera by r -1 After that, the two image planes are coplanar. Solve and calculate the rotation matrix R w =[e1 e2 e3] T , where e1 is the unit vector in the baseline direction, e2 is the orthogonal vector between the unit vector e1 and the principal optical axis, and e3 is the direction vector of the coordinate axis in the camera coordinate system. e1, e2, and e3 can be expressed as:

[0055]

[0056] In the formula, T is the translation vector. Combine the coplanar matrix and the row alignment matrix to obtain the left eye matrix R l =R w r and the right matrix R r =R w r -1 And act on the image to complete the extreme correction of the binocular image. Stereo matching is performed using the SAD descriptor as the matching cost, and the multi-directional scanline optimization (Scanline Optimizer, SO) algorithm based on one-dimensional dynamic programming can effectively reduce the matching cost. The optimization process of the cost p in the direction r under the cost image C(d) can be expressed as:

[0057]

[0058] Where C(p, d) is the original cost of the pixel in the unaggregated state; the second term on the right side of the formula is the smoothing term, Cr (pr, d) is the matching cost of the previous pixel in the r direction, P1 is the penalty value when the change of the disparity d of the previous pixel in the r direction is ±1, and P2 is the penalty value when the change of the disparity d of the previous pixel in the r direction is greater than 1. The minimum value of the second term in the formula reflects the difference between the original cost C(p, d) and the neighborhood, and the third term on the right side of the formula constrains the upper limit of the matching cost. The integrated controller converts the acquired disparity image into a point cloud image according to the internal and external parameters of the binocular infrared camera.

[0059] 2) Target recognition and segmentation

[0060] For point cloud target extraction, the most direct method is to perform it on the point cloud, but due to the complexity of 3D point cloud data, direct extraction will take a long time and it is difficult to achieve real-time performance. Therefore, we use 2D target extraction and then project it into 3D space to achieve 3D target extraction. In order to achieve 2D target extraction, this paper uses the Mask R-CNN model to segment and extract targets from 2D images. The Mask R-CNN model is developed from the Faster R-CNN model. It is a convolutional neural network (CNN) that can return a bounding box and its category label and confidence score for each object. The model can generate a bounding box and segmentation mask for each object instance in the image. The model uses a feature pyramid network (FPN) and a ResNet backbone network.

[0061] The Mask R-CNN network training is implemented as follows:

[0062] 1. Data set annotation and production

[0063] Use polygons to circle the labelme of the object to be extracted to create a data set, and use the opencvdrawContours function to generate a mask binary graphic area based on the mask polygon points of the data set.

[0064] 2. Data augmentation

[0065] In the loaddata stage, random rotation and flipping are used in the geometric space, and random changes in brightness and contrast are used in the color space. The changes in the alpha and beta coefficients are combined to integrate the changes in the set space and the color space and then applied to the image to complete data augmentation.

[0066] 3. Target extraction

[0067] The confidence of the target box is set to 0.6, the msk threshold is set to 0.8, all detection boxes with confidence greater than 0.6 are marked, and masks are generated for areas in the target box where the msk threshold is greater than 0.8 to complete the target extraction of the image.

[0068] 3) Pixel alignment and point cloud segmentation

[0069] In order to complete the fusion of the target tool’s image feature map G and point cloud feature map Q, we first need to solve the alignment relationship between each point and each pixel in the point cloud map. 3×4 The external parameter matrix between the camera Project each point of Q into the coordinate system of G:

[0070]

[0071] Where, d i is the depth scale factor; [u i v i 1] T is the homogeneous coordinate of the projection of the point in the graph Q onto the coordinate system of the graph G; s is the scaling ratio of the original image to the graph G; [x i y i z i 1] T is the three-dimensional homogeneous coordinate in the point cloud feature map Q.

[0072] 4) Posture measurement

[0073] The pose measurement is performed on the basis of 3D point cloud, and the 6D transformation relationship from source point cloud (source) to reference point cloud (model) is calculated. The reference point cloud is collected offline by a high-precision laser line scanning 3D camera. After point cloud registration and manual refinement, the obtained point cloud has high quality. The source point cloud is the segmented target point cloud collected in real time.

[0074] Point cloud registration can be achieved in two steps: coarse registration and fine registration. The algorithm flow is as follows: Figure 3 As shown. Coarse registration first needs to calculate the SHOT feature descriptor of the point cloud, extract the key points of the source and model, and calculate the rough conversion relationship between the two point clouds S. After coarse registration, an initial matrix can be obtained, so that the two point clouds fit to a relatively good position, but generally do not completely overlap. At this time, the point clouds need to be finely registered. Fine registration uses the pose calculated by coarse registration as the initial value, and is implemented using the ICP algorithm to finally complete the pose calculation of the target tool. The point cloud feature calculation and coarse registration steps are implemented using the CPU, while the fine registration ICP algorithm is implemented using the GPU. The pose calculation process is as follows: Figure 4 shown.

[0075] In summary, the high-precision three-dimensional point cloud measurement system of the present invention adopts binocular and speckle structured light to realize the generation of three-dimensional point cloud. The speckle projection unit projects speckle to the target area. After the binocular camera forms an image, the imaging data is processed by the integrated controller to generate point cloud data. The recognition and posture calculation of the target tool are completed through the algorithm. It is a multi-target, high-precision, stable, fast and strong anti-interference three-dimensional point cloud measurement system, which can be used in environments with complex lighting.

[0076] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A high-precision three-dimensional point cloud measurement system, characterized in that: It includes: a front-end image detection device, an integrated controller and a host computer connected in sequence; Among them, the front-end image detection equipment includes: A speckle projection unit is arranged in the middle of the front-end image detection device and is used to project speckles onto the target; Two near-infrared cameras are respectively set at the two ends of the front-end image detection device to perform binocular measurement and structured light measurement of the target; The observation camera is set in the middle of the front-end image detection device to realize imaging of the target within the visible light range; The front-end image detection equipment is used to project speckles onto the target, collect real-time images on site, receive and execute instructions sent by the host computer, and transmit image data to the integrated controller; The integrated controller is used to receive images from the front-end image detection equipment, pre-process the point cloud generation algorithm, synchronize the camera, communicate, and control the speckle projection; The host computer is used to control the entire system and display the system's working status, and transmit the target posture information to the front-end image detection device.

2. The high-precision three-dimensional point cloud measurement system according to claim 1, characterized in that: The front-end image detection equipment is installed on the robot.

3. The measurement method of the high-precision three-dimensional point cloud measurement system according to claim 1 or 2, characterized in that: The following steps are involved: Step 1: After the two near-infrared cameras collect image information, they use the SGBM algorithm for stereo matching to solve the disparity of each pixel and convert the disparity image into a point cloud image; Step 2: Obtain RGB images through the observation camera and extract the target area image. Use the Mask R-CNN network to segment the color image. Determine the alignment relationship between each 3D point and each pixel through pixel alignment to complete the fusion of the image feature map and the point cloud feature map. Step 3: Use the PnP algorithm and ICP algorithm to achieve coarse and fine registration of the point cloud map, complete the pose calculation of the target tool, and guide the robot to grasp and operate the tool.

4. The measuring method according to claim 3, characterized in that: Step 1 specifically includes: The Zhang Zhengyou calibration method is used to calibrate the internal and external parameters and distortion parameters of the near-infrared camera and observation camera to ensure the measurement accuracy of the measurement system; The speckle projection unit projects speckles onto the target area, and two near-infrared cameras collect binocular images and transmit them to the integrated controller; The integrated controller performs epipolar correction on the acquired binocular images, and then uses the SGBM algorithm for stereo matching to solve the disparity of each pixel and convert the disparity image into a point cloud image.

5. The measuring method according to claim 3, characterized in that: Step 2 specifically includes: The observation camera acquires the RGB image of the target and transmits it to the integrated controller, which extracts the target area image and uses the Mask R-CNN network to segment the color image. The integrated controller determines the alignment relationship between each 3D point and each pixel by pixel alignment, and completes the fusion of the image feature map and the point cloud feature map.