A binocular camera positioning method for a satellite antenna operation robot in a tunnel

Abstract

The present invention discloses a binocular camera positioning method for a satellite antenna operating robot in a tunnel. First, the internal and external parameters of the binocular camera are calibrated to generate a binocular camera model. The "row alignment" operation is achieved through image adjustment technology. The position coordinates of the satellite antenna feature points in the image are combined with the parallax values ​​of the coordinates in the left and right images. The internal and external parameters of the head-up binocular camera model are used to calculate the depth information of the satellite antenna and the coordinates in the world coordinate system including the X, Y, and Z directions. The present invention can reduce the system impact caused by radial distortion and tangential distortion; adopt Zhang Zhengyou's calibration dot matrix method to cover the full field of view of the binocular camera through multi-image calibration; adopt the method of maximizing the common area of ​​the image to minimize the distortion caused by reprojection and maximize the common area; and have the characteristics of translation / rotation invariance, scale invariance, overlap invariance, light source change invariance, contrast flip invariance, etc.

Description

Technical Field

[0001] The present invention relates to the fields of image processing and machine vision, designs NI Vision software, and specifically relates to a binocular camera vision system used to obtain the position positioning of a satellite antenna operating robot in a tunnel. Background Art

[0002] Satellite navigation technology is widely used in space systems such as mountains and tunnels in my country's complex terrain. With the improvement of automation level, the use of machine vision methods can effectively improve the adaptability of positioning in tunnels, effectively increase the operating range, and effectively avoid the unstable accuracy and quality caused by manual visual inspection.

[0003] Binocular cameras are the smallest stereo vision systems, capable of completing most 3D tasks. They capture images of the target using two cameras, align them to the same plane, and then reconstruct 3D information based on the parallax between pixels corresponding to feature points. Binocular vision transforms the 2D image coordinate system into a 3D world coordinate system, which is then transmitted to the robot for automated positioning and task completion.

[0004] The patent "Light Strip-Based Tunnel Positioning System and Distance Measurement Method CN113514855A" proposes a tunnel positioning system method based on light strips, which mainly includes multiple sets of light strips distributed in the tunnel area; a power driver is used to drive the switches of the light strips; a controller is connected to the power driver to output a control signal to control the light strips; multiple locators are matched with the light strips, each locator is used to obtain a positioning signal of the device to be located; and an automatic positioning component is used to install the light strips in the tunnel in conjunction with the light strips, and the automatic positioning component includes a light strip positioning block, a tunnel positioning block and the light strips. This method can effectively achieve positioning in the tunnel, but it requires the pre-laying of light strips and other devices in the tunnel, which is costly and labor-intensive. It cannot be completed in tunnels with small spaces and inaccessible to personnel, and its scope of use is limited.

[0005] The patent "Binocular Camera and Distance Measurement Method Based on Binocular Camera CN106911923A" proposes a binocular camera that mainly includes a first imaging lens, a second imaging lens, a first image sensor, a second image sensor, and a beam splitter. The beam splitter is used to split a single light beam entering the binocular camera into two second light beams that are projected onto the second imaging lens. The reflective surface in the light path of the light beam is mirrored to a symmetrical surface. In the equivalent light path formed, the optical axes of the first imaging lens and the second imaging lens are parallel to each other and offset from each other by a predetermined distance. This method uses a beam splitter to collect data, which can reduce the problem of image occlusion between the two lenses relative to each other. However, the measurement accuracy increases due to the addition of a mechanical structure link, resulting in low system measurement accuracy.

[0006] In order to solve the shortcomings of the above algorithms, the system designed in this paper needs to use relevant technologies such as Zhang Zhengyou calibration dot matrix method, polynomial distortion calibration model, head-up binocular camera model, image adjustment method that maximizes image common area, and edge-based geometric matching positioning algorithm. Summary of the Invention

[0007] In view of this, the purpose of the present invention is to meet the application of satellite antennas in tunnels with narrow spaces, and to solve problems such as manual work being impossible, low efficiency, and high cost. The technical idea of ​​the present invention is: to use a method based on a binocular camera to provide positioning for satellite antennas in tunnels.

[0008] First, the internal and external parameters of the binocular camera are calibrated to generate a binocular camera model. The "row alignment" operation is implemented through image adjustment technology. The position coordinates of the satellite antenna feature points in the image are combined with the disparity values ​​of the coordinates in the left and right images. The internal and external parameters of the head-up binocular camera model are used to calculate the depth information of the satellite antenna, thereby calculating the coordinates in the world coordinate system including the X, Y, and Z directions. The specific implementation steps are as follows:

[0009] Step 1: Prepare the binocular camera environment;

[0010] Step 1.1: Select a binocular camera, light source, lens, and computer. Install the binocular camera at a suitable height and angle on the robot support. Also install a light source of appropriate intensity. During installation, ensure that the optical axis is parallel and perpendicular to the baseline. The baseline is the line connecting the optical centers of the two industrial cameras.

[0011] Step 1.2: Use the calibration dot matrix to calibrate the intrinsic and extrinsic parameters of the binocular camera;

[0012] Step a: Establish models of an irregular binocular vision system and a head-up binocular vision system. Based on the geometric model, calculate the three-dimensional coordinates. It is necessary to know the intrinsic parameters, extrinsic parameters and parallax of the homologous points of the camera. Establish the camera coordinate system, world coordinate system and image coordinate system of the binocular camera system. Set the coordinate system of the world coordinate system to coincide with the coordinate system of the left camera.

[0013] Step b: Print the dot matrix calibration plate according to the dot matrix calibration method provided by Zhang Zhengyou, set the parameters of the binocular camera calibration template, and then use the binocular camera to collect multiple angles and multiple pairs of dot matrix calibration plates. Calibrate the left and right binocular vision systems in the IMAQ Learn Binocular Stereo Calibrator VI provided by LabVIEW. First, perform single-camera calibration using internal parameters such as the focal length, optical center, and polynomial distortion model of the left and right cameras, as well as external parameters such as the rotation and translation relationship between the camera coordinate system and the world coordinate system; then calibrate the binocular system to obtain the position rotation and translation relationship between the cameras;

[0014] Step c: after obtaining the calibration coefficients, generate a binocular camera model, and based on this model, collect binocular camera images while the robot moves forward;

[0015] Step 1.3, adjusting the collected image using the maximum image area algorithm;

[0016] Step a: Use binocular cameras to simultaneously take photos of the tunnel environment to be inspected, including the satellite antenna. At this time, the left and right cameras take photos simultaneously, ensuring that the same object is photographed at the same time, thereby ensuring the accuracy of the homology point.

[0017] Step b: Using the system internal parameters, external parameters, and relative position relationship obtained in step 1.2, and the algorithm proposed by Jean-Yves Bouguet to maximize the common area of ​​the image, the rotation matrix is ​​decomposed. This can rotate the left and right cameras by half an angle to make the optical centers parallel.

[0018] Step c: After step b is completed, construct a transformation matrix so that the baseline is perpendicular to the optical axes of the left and right camera images;

[0019] Step d: After step c is completed, the left and right cameras are rotated to implement the "row alignment" operation, thereby converting the irregular binocular camera into a head-up binocular camera model;

[0020] Step 1.4: Preprocessing using low-pass filter and pixel comparison algorithm

[0021] Step a: After the adjusted image is obtained after step 1.3, a low-pass filter is used to filter the image to reduce noise in the image;

[0022] Step b: for the filtered image, pixel comparison is performed again by setting a comparison operator, and the background part of the pixel value is set to 0 to improve the matching accuracy;

[0023] Step 2: Perform pattern matching based on geometric matching algorithm;

[0024] Step a: Based on the geometric matching algorithm, the left camera image adjusted and preprocessed in step 1.4 is used to learn the satellite antenna feature point contour, perform edge detection, and extract the satellite antenna contour in the view;

[0025] Step b: Learning the template. After the algorithm extracts the edge curve from the image, it calculates the grayscale gradient direction angle of each pixel on the edge curve and normalizes all edges with the same gradient direction angle.

[0026] Step c: Capture images while the robot is moving forward, turn on the light source, perform image adjustment and preprocessing based on step 1.4, and perform a global search on the adjusted image based on the geometric edges of the learned edge image template. If feature points can be matched to both images, the coordinates of the homologous points can be obtained.

[0027] Step 3: Calculate the world coordinate system based on the homologous pixel coordinates of the feature points

[0028] Step a: After completing step 2, the pixel coordinates of the homologous points are obtained, and the disparity map of the pixel points can be obtained. The depth information of the binocular camera is calculated through the mathematical model of the head-up binocular vision system, as well as the intrinsic parameters, extrinsic parameters and relative position relationship obtained after calibration;

[0029] Step b: Calculate the X, Y, and Z coordinates of the feature points in the world coordinate system based on the mapping relationship of the geometric model;

[0030] Step c: sending the obtained three-dimensional coordinates of the satellite antenna to the mobile robot, and having the robot perform the operation;

[0031] This invention implements the binocular camera's positioning function using depth information, enabling the calculation of the satellite antenna's position within a tunnel. It uses a polynomial distortion model to effectively reduce the system impact of radial and tangential distortion. It employs Zhang Zhengyou's calibration dot matrix method, which allows for multi-image calibration to cover the binocular camera's full field of view. It also employs a method for maximizing the common area of ​​images to minimize distortion caused by reprojection and maximize the common area. Template matching uses an algorithm based on geometric matching, which, compared to grayscale matching, exhibits translational / rotational invariance, scale invariance, overlap invariance, invariance to light source changes, and invariance to contrast reversal. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is the workflow diagram of the binocular camera.

[0033] Figure 2 Schematic diagram of the calibration of the head-up binocular camera.

[0034] Figure 3 Collect image library for stereo camera calibration.

[0035] Figure 4 Adjust the effect for the image from the binocular camera.

[0036] Figure 5 This is the edge geometry effect of the binocular camera. DETAILED DESCRIPTION

[0037] The following is a detailed description of the workflow of this system with reference to the accompanying drawings and specific implementation cases.

[0038] Reference Figure 1 , the workflow steps of this system are as follows:

[0039] Step 1: Calibrate the binocular camera's intrinsic and extrinsic parameters and generate a camera model in LabVIEW. The steps are as follows:

[0040] Step 1.1: Establish the world coordinate system, camera coordinate system, and image coordinate system of the binocular camera. Define the world coordinate system as the left camera coordinate system. The camera coordinate system is a three-dimensional coordinate system with the camera optical center as the origin and the optical axis as the Z axis. The left and right camera coordinate systems can be transformed by rotation and translation. The image coordinate system is a coordinate system established based on the image pixels of the left and right cameras.

[0041] Step 1.2: Establish the relationship between pixel coordinate transformation from the world coordinate system to the image coordinate system:

[0042]

[0043] in, and are the lengths corresponding to unit pixels, f is the focal length of the camera, R is the rotation matrix of the right camera relative to the left camera, and B is the translation matrix of the right camera relative to the left camera.

[0044] Point P in the world coordinate system W (X w , Y w , Z w ) to the image coordinate system of the left camera can be directly calculated through the triangle projection relationship:

[0045]

[0046] The subscript L represents the value related to the left camera. The position of the point in the right camera image coordinate system can also be calculated by projection transformation. However, since the projection transformation must be performed in the corresponding camera coordinate system, the world coordinates P of the spatial point must be calculated first. W (X w , Y w , Z w ) is transformed into the coordinate P in the right camera coordinate system by rotation and translation R (X R , Y R , Z R ) is calculated and expressed as follows:

[0047]

[0048] Therefore, four equations are obtained through the projection transformation relationship between the two cameras to calculate the world coordinates P of the spatial point W (Xw , Y w , Z w ).

[0049] Step 1.3: Open the image acquisition interfaces of the left and right cameras in LabVIEW at the same time, set the parameters of the calibration template, and continuously take 12 sets of calibration plate images to calibrate the entire field of view. Figure 3 ;

[0050] Step 1.4: Use the IMAQ Learn Camera Model program to calibrate the left and right cameras' internal parameters such as focal length, optical center, and distortion model, as well as the external parameters such as rotation and translation between the left and right cameras and the world coordinate system. Use the IMAQ LearnBinocular Stereo Calibration program to calibrate the binocular camera and obtain the rotation and translation relationship between the camera coordinate system and the other camera coordinate systems. Figure 2 .

[0051] Step 2. Use IMAQ Get Rectfied Image From Stero.vi in ​​LabVIEW to adjust the left and right images. Set Interpolation Type to Bi-Linear to achieve "row alignment" operation. Figure 4 , the steps are as follows:

[0052] Step 2.1 uses the algorithm proposed by Bouguet's method to maximize the common area of ​​the images. After the system is calibrated, the rotation matrix R and the translation matrix T are decomposed to rotate the left and right cameras by half so that the optical centers are parallel:

[0053] Then R r =R 1 / 2 , R l =R -1 / 2

[0054] Step 2.2: Construct a transformation matrix so that the baseline is parallel to the imaging plane. The transformation matrix is ​​defined as follows:

[0055]

[0056] in:

[0057] The unit direction vector of the left pole in the left image when the projection center is the image coordinate origin, and T is the translation vector.

[0058] It is the cross product of e1 and the unit vector in the direction of the optical axis.

[0059] e3=e1×e2, that is, e1, e2, and e3 are orthogonal.

[0060] Step 2.3 rotates and aligns the left and right images. The transformation matrix is ​​defined as follows:

[0061] R L =R rectify R r

[0062] R R =R rectify R l

[0063] Step 3: Use the image pixel-by-pixel comparison method for preprocessing. The steps are as follows:

[0064] Step a: After the adjusted image is obtained in step 2.3, use the low-pass filter IMAQ LowPass.vi to filter the adjusted image to reduce the noise in the image.

[0065] Step b: For the filtered image, perform pixel comparison again using the IMAQ Compare.vi comparison operator, setting the Operator parameter to "Clear if < or =" and the threshold to 240, which can set the background portion of the pixel value to 0;

[0066] Step 4: Use the edge curve-based geometric matching method to find pixel disparity, refer to Figure 5 , the steps are as follows:

[0067] Step 4.1: Use the ROI tool to select the smallest feature image containing the satellite antenna in the image as a learning template for geometric template matching;

[0068] In step 4.2, use IMAQ Advanced Setup Learn Geometric Pattern (Feature Based).vi in ​​LabVIEW to learn the pattern. Set the Setup Data Item to Minimum Scale and the ItemValue to 80.

[0069] In step 4.3, use IMAQ Match Geometric Pattern (Feature Based).vi to perform a full-image search of the adjusted left and right images. Set the Number of Matches to 1 and the Minimum Score to 800 to obtain the pixel coordinates, rotation angle, and score of the feature points.

[0070] Step 4.4, calculate the pixel coordinates p(u L , v L ) and p(u R , v R ), calculate the disparity u of the feature points L -u R ;

[0071] Step 5: Obtain binocular calibration parameters to calculate the world coordinate system position

[0072] Step 5.1 Use IMAQ Get Binocular Stereo Calibration Info.vi to obtain the internal and external parameters of the binocular camera model. It can directly output the Q matrix, which is the transformation matrix used to convert the pixel coordinate system to the world coordinate system.

[0073] Step 5.2 According to step 4.4, p(u L , v L ) and p(u R , v R ) and the Q matrix obtained in step 5.1, the three-dimensional coordinates of the feature points can be calculated as follows:

[0074]

Claims

1. A binocular camera positioning method for a satellite antenna operation robot in a tunnel, characterized by: First, the internal and external parameters of the binocular camera are calibrated to generate a binocular camera model. Image adjustment technology is used to achieve "row alignment" operations. The position coordinates of the satellite antenna feature points in the image are combined with the disparity values ​​of these coordinates in the left and right images. Using the internal and external parameters of the head-up binocular camera model, the depth information of the satellite antenna is calculated, thereby calculating the coordinates in the world coordinate system including the X, Y, and Z directions. The specific implementation steps are as follows: Step 1: Prepare the binocular camera environment; Step 1.1: Select a binocular camera, light source, lens, and computer. Install the binocular camera at a suitable height and angle on the robot support. Also install a light source of appropriate intensity. During installation, ensure that the optical axis is parallel and perpendicular to the baseline, which is the line connecting the optical centers of the two industrial cameras. Step 1.2: Use the calibration dot matrix to calibrate the intrinsic and extrinsic parameters of the binocular camera; Step 1.3, adjusting the collected image using the maximum image area algorithm; Step 1.4: Preprocessing using a low-pass filter and pixel comparison algorithm; Step 2: pattern matching based on geometric matching algorithm; Step a: Based on the geometric matching algorithm, the left camera image adjusted and preprocessed in step 1.4 is used to learn the satellite antenna feature point contour, perform edge detection, and extract the satellite antenna contour in the view; Step b: Learning the template. After the algorithm extracts the edge curve from the image, it calculates the grayscale gradient direction angle of each pixel on the edge curve and normalizes all edges with the same gradient direction angle. Step c: Capture images while the robot is moving forward, turn on the light source, perform image adjustment and preprocessing based on step 1.4, and perform a global search on the adjusted image based on the geometric edges of the learned edge image template. If feature points can be matched to both images, the coordinates of the homologous points can be obtained. Step 3: Calculate the world coordinate system based on the homologous pixel coordinates of the feature points; Step a: After completing step 2, the pixel coordinates of the homologous points are obtained, and the disparity map of the pixel points can be obtained. The depth information of the binocular camera is calculated through the mathematical model of the head-up binocular vision system, as well as the intrinsic parameters, extrinsic parameters and relative position relationship obtained after calibration; Step b: Calculate the X, Y, and Z coordinates of the feature points in the world coordinate system based on the mapping relationship of the geometric model; Step c: Send the obtained three-dimensional coordinates of the satellite antenna to the mobile robot, and let the mobile robot perform the operation.

2. The binocular camera positioning method for a satellite antenna operating robot in a tunnel according to claim 1, characterized in that: Step 1.2 includes, Step a: Establish models of an irregular binocular vision system and a head-up binocular vision system. Based on the geometric model, the three-dimensional coordinates need to be calculated by knowing the intrinsic parameters, extrinsic parameters, and parallax of the homologous points of the camera. Establish the camera coordinate system, world coordinate system, and image coordinate system of the binocular camera system, and set the world coordinate system to coincide with the left camera coordinate system. Step b: Print the dot matrix calibration plate according to the dot matrix calibration method provided by Zhang Zhengyou, set the parameters of the binocular camera calibration template, and then use the binocular camera to collect multiple pairs of multi-angle dot matrix calibration plates. Calibrate the left and right binocular vision systems in the IMAQLearn Binocular Stereo Calibrator VI provided by LabVIEW. First, perform a single camera calibration using the focal length, optical center, and polynomial distortion model internal parameters of the left and right cameras, as well as the rotation and translation relationship external parameters between the camera coordinate system and the world coordinate system; then calibrate the binocular system to obtain the position rotation and translation relationship between the cameras; Step c: After obtaining the calibration coefficients, a binocular camera model is generated, and binocular camera images are collected based on this model while the robot is moving forward.

3. The binocular camera positioning method for a satellite antenna operating robot in a tunnel according to claim 1, characterized in that: Step 1.3 includes, Step a: Use binocular cameras to simultaneously take photos of the tunnel environment to be inspected, including the satellite antenna. At this time, the left and right cameras take photos simultaneously, ensuring that the same object is photographed at the same time, thereby ensuring the accuracy of the homology point. Step b: Using the system intrinsic parameters, extrinsic parameters, and relative position relationship obtained in step 1.2, and the algorithm proposed by Jean-Yves Bouguet to maximize the common area of ​​the image, decompose the rotation matrix and rotate the left and right cameras by half an angle to make the optical centers parallel; Step c: After step b is completed, construct a transformation matrix so that the baseline is perpendicular to the optical axes of the left and right camera images; Step d: After step c is completed, the left and right cameras are rotated to achieve the "row alignment" operation, thereby converting the irregular binocular camera into a head-up binocular camera model.

4. The binocular camera positioning method for a satellite antenna operating robot in a tunnel according to claim 1, characterized in that: Step 1.4 includes, Step a: After the adjusted image is obtained after step 1.3, a low-pass filter is used to filter the image to reduce noise in the image; Step b: for the filtered image, pixel comparison is performed again by setting a comparison operator, and the background part of the pixel value is set to 0 to improve the matching accuracy.

Images