A highway pile number matching method based on binocular stereo vision

Abstract

The present invention discloses a method for matching highway pile numbers based on binocular stereo vision. The method comprises the following steps: applying binocular stereo vision principles to process road space and image information collected through a vehicle-mounted multi-frequency, multi-system combined inertial navigation system and a binocular camera; processing the road positioning data using a polynomial interpolation method to obtain a driving trajectory; identifying and extracting mileage pile numbers from the image data, naming and storing the image data according to the mileage pile number; matching the pixels of the upper left corner of the mileage pile number plate in images corresponding to the same mileage pile; converting the pixel coordinates into three-dimensional coordinates in a camera coordinate system with the left camera as the origin, combining the driving trajectory to obtain position information, combining the position information with the rotation and translation relationship to obtain a reprojection matrix, and then obtaining the geodetic coordinates corresponding to the mileage pile. The present invention achieves high-precision, rapid, and automated matching of highway pile numbers and geodetic coordinates, facilitating rapid positioning for road maintenance and management work.

Description

Technical Field

[0001] The invention belongs to the technical field of highway traffic management, and in particular relates to a highway pile number matching method based on binocular stereo vision. Background Art

[0002] With the rapid development of expressway construction in my country, road management and maintenance are becoming increasingly important. In the era of digital transportation, road management and maintenance are gradually shifting towards digitalization and intelligence. Currently, digital data collection for road infrastructure often uses longitude and latitude coordinates as the spatial measurement scale. However, actual road construction and maintenance work is based on highway mileage stake numbers, with design stake numbers used during the design phase and operational stake numbers used during maintenance. Therefore, matching longitude and latitude coordinates with highway stake numbers is essential. However, existing methods for matching highway stake numbers with three-dimensional coordinates lack intelligence and accuracy, resulting in a lack of efficiency and accuracy that meets practical application requirements. Summary of the Invention

[0003] The purpose of the present invention is to provide a highway pile number matching method based on binocular stereo vision to solve the problems existing in the above-mentioned prior art.

[0004] To achieve the above object, the present invention provides a highway pile number matching method based on binocular stereo vision, comprising:

[0005] A navigation system and a binocular camera are installed on a data collection vehicle, the rotation and translation relationship is recorded, and the binocular camera is calibrated and stereo corrected; the data collection vehicle collects road positioning data and image data along the road section to be matched, and the image data includes left camera image data and right camera image data;

[0006] Processing the road positioning data using a polynomial interpolation method to obtain a driving trajectory of the data collection vehicle;

[0007] Identify and extract the mileage post number in the image data, name and store the image data according to the mileage post number, and establish a left camera mileage post image database and a right camera mileage post image database respectively;

[0008] Perform pixel matching on the images of the same mileage post in the left camera mileage post image database and the right camera mileage post image database to obtain disparity and calculate the depth of field, where the pixel position is at the upper left corner of the mileage post number plate;

[0009] According to the parallax, the depth of field and the geometric relationship, the pixel coordinates are converted into three-dimensional coordinates in a camera coordinate system with the left camera as the origin, and the posture information is obtained in combination with the driving trajectory; based on the posture information and the rotation and translation relationship, a reprojection matrix of the camera coordinate system relative to the world coordinate system is obtained when a number of image data are acquired, and based on the reprojection matrix, the three-dimensional coordinates in the camera coordinate system are converted into geodetic coordinates corresponding to the milepost.

[0010] Optionally, the calibration process of the binocular camera includes: calibrating each camera in the binocular camera one by one using the Zhang Zhengyou calibration method; setting up and fixing relatively parallel left and right cameras, shooting a checkerboard image, establishing a coordinate system with the optical center of the left camera as the origin, and combining the shot checkerboard image to obtain the rotation matrix and translation matrix of the right camera relative to the left camera to complete the calibration of the binocular camera.

[0011] Optionally, the road positioning data includes: longitude and latitude, elevation, three-way vehicle speed, yaw angle, pitch angle and roll angle.

[0012] Optionally, the process of obtaining the driving trajectory includes:

[0013] The vehicle's motion is decomposed into three directions, including east, north, and sky. Cubic polynomial interpolation is performed on the three directions of motion respectively to fit the continuous motion trajectory between the path points in the three directions. Motion synthesis is then performed to obtain the driving trajectory of the data collection vehicle, where the path points are points that include spatial information and posture information.

[0014] Optionally, the process of performing cubic polynomial interpolation on the three directions of motion includes: taking adjacent path points in the three directions as starting points and end points, respectively, and constructing a trajectory function, a velocity function, and an acceleration function between the adjacent points;

[0015] Obtaining constraints of trajectory functions and speed functions in three directions respectively based on the road positioning data obtained by the navigation system at different times;

[0016] The trajectory function is solved according to the constraints to obtain the unknown coefficients of the trajectory function, and then the position and speed of the vehicle at any time between two adjacent trajectory points in unidirectional motion are obtained, and the motion interpolation results in three directions are obtained respectively.

[0017] Optionally, the process of identifying and extracting the mileage station number includes:

[0018] Create an HSV color component adjuster, set the HSV color component range according to the characteristics of highway mileage post number plates, convert the image data into an HSV image, adjust the HSV image using the HSV color component adjuster, perform preliminary positioning of the adjusted image in the HSV color space, extract all contours in the preliminary positioning result, filter the contours according to the aspect ratio of the mileage post number plates, and perform character segmentation on the corresponding areas in the image to obtain character segmentation areas; obtain a standard image of the mileage post number plate, obtain a standard digital template based on the standard image, calculate the matching score of the character segmentation area corresponding to each mileage post and the standard digital template, take the optimal solution and output it in the character segmentation order, rename the image data according to the output template content, and store it in the mileage post image information database.

[0019] Optionally, a sliding window method is used for pixel matching, the pixel window is limited to a square with a fixed side length, and a normalized correlation coefficient is used to determine the correlation between the pixel to be matched and the candidate pixel.

[0020] The technical effects of the present invention are:

[0021] The present invention proposes a highway pile number matching method based on binocular stereo vision, aiming to achieve high-precision, rapid, and automated matching of highway pile numbers and geodetic coordinates. The method applies the principle of binocular stereo vision to collect road space and image information through a vehicle-mounted multi-frequency, multi-system combined inertial navigation system and a binocular camera, obtaining the correspondence between mileage pile numbers and geodetic coordinates with an error of decimeter level. This method realizes the rapid conversion between pile numbers and longitude and latitude, facilitating rapid positioning for road maintenance and management work. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0023] Figure 1 Schematic diagram of the method flow in an embodiment of the present invention;

[0024] Figure 2 Schematic diagram of the position of each vehicle-mounted device in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] Example 1

[0028] like Figure 1-2 As shown, this embodiment provides a highway pile number matching method based on binocular stereo vision, including:

[0029] S1. Install a multi-frequency, multi-system combined navigation system and a binocular camera on the data collection vehicle, record the positional relationship between the devices, perform camera calibration and stereo correction on the binocular camera, and drive along the road section to be matched to collect road positioning data and image data.

[0030] S2. Using a cubic polynomial interpolation method to process the positioning data collected by the integrated navigation system and fit the driving trajectory of the data collection vehicle;

[0031] S3. Using image recognition technology to identify and extract the mileage pile numbers from the image data collected by the left and right cameras of the binocular camera, and naming the image data according to the pile numbers to establish a mileage pile image database for the left camera and the right camera;

[0032] S4. Use the sliding window method to match the pixels of the upper left corner of the mileage post number plate of the images corresponding to the same mileage post in the mileage post image database of the left camera and the right camera, and calculate the depth of field;

[0033] S5. Convert pixel coordinates into three-dimensional coordinates in the camera coordinate system with the left camera as the origin through geometric relationships. Use trajectory fitting to obtain the position information and the rotation and translation relationship of the equipment setup to obtain the reprojection matrix of the camera coordinate system relative to the world coordinate system at the moment the milepost image is captured. Use the reprojection matrix to convert the three-dimensional coordinates in the camera coordinate system into the geodetic coordinates of the highway milepost.

[0034] Step S1 is specifically as follows:

[0035] S1-1. The vehicle-mounted multi-frequency multi-system integrated navigation system and the vehicle-mounted binocular camera are installed on the data collection vehicle. Figure 2 As shown, the rotation and translation relationship between the camera coordinate system of the left camera of the binocular camera and the inertial navigation coordinate system of the integrated navigation system is the translation matrix T1 and the rotation matrix R1, and the rotation and translation relationship between the inertial navigation coordinate system of the integrated navigation system and the vehicle is the translation matrix T2 and the rotation matrix R2. The vehicle is generally a data collection vehicle.

[0036] S1-2. Camera calibration includes single camera calibration and binocular camera calibration. Single camera calibration uses Zhang Zhengyou calibration method. The camera to be calibrated takes a set of no less than 20 images of a checkerboard of known size from different angles. The checkerboard corners in the image are detected using Python code to obtain the pixel coordinates of the corners. Based on the known world coordinates, a relationship matrix is ​​established to solve the intrinsic parameter matrix and radial distortion coefficient.

[0037] S1-3. Binocular camera calibration: Use the MATLAB built-in application "Stereo Camera Calibrator" to calibrate. Set up and fix two relatively parallel cameras to shoot a checkerboard. Establish a coordinate system with the optical center of the left camera as the center. Solve the rotation matrix R and translation matrix T of the right camera relative to the left camera.

[0038] S1-4. The data collection vehicle should drive in the lane closest to the pile number sign during data collection, and avoid collecting data during precipitation.

[0039] S1-5. The protocol for exporting data from the integrated navigation device is GPCHC, which includes latitude and longitude, elevation, three-way vehicle speed, yaw angle, pitch angle, and roll angle.

[0040] Step S2 is specifically as follows:

[0041] S2-1. Decompose the vehicle's motion into three directions: east, north, and sky. E i and As the position and velocity of the eastward motion, N i and As the position and velocity of the northward motion, H i and V hi As the position and speed of the celestial motion. Perform cubic polynomial interpolation on the three-way motion, fit the continuous motion trajectory between the three-way path points, and finally perform motion synthesis to obtain the driving trajectory of the captured vehicle;

[0042] Waypoints are points containing spatial information, including position and posture information, collected by the navigation system. These waypoints do not correspond to the shooting points. Trajectory fitting is used to obtain the time-corresponding driving trajectory, and the corresponding shooting location is determined by the image data shooting time.

[0043] S2-2. In the three-way motion, the adjacent path points in each direction are regarded as the starting point and the end point. That is, the positions of the endpoints of all trajectory fitting intervals are known, and the trajectory function between adjacent points is S(t) = a0 + a1t + a2t 2 +a3t 3 , the speed function is The acceleration function is Where t is the time of the trajectory point, a0, a1, a2, a3 are the coefficients to be determined;

[0044] S2-3. In the three-way motion, the trajectory function in each direction satisfies the endpoint position constraints S(0) = S0, S(t f )=S f and endpoint velocity constraints The constraints can be confirmed by the position and velocity data of the data points collected by the integrated navigation system;

[0045] S2-4, according to the constraint conditions, the unknown coefficient in the trajectory function is a0=S0, The goal of this step is to solve the trajectory function. The other two functions are derivatives of the trajectory function and have the same coefficients. Therefore, solving the trajectory function naturally leads to the other two functions, which serve to aid in solving the problem by combining constraints. A cubic polynomial is determined using the undetermined coefficients to describe the vehicle's position and speed at any moment between two adjacent trajectory points during unidirectional motion. The same polynomial is used for the same trajectory segment, resulting in motion interpolation. Finally, the three-dimensional trajectory interpolation result is vector synthesis, which is the captured vehicle's trajectory.

[0046] Step S3 is specifically as follows:

[0047] S3-1. Create an HSV color component adjuster.

[0048] Since the uniform background color of the mileage post number plates on my country's expressways is dark green, the HSV color component range is initially set to H∈[80, 120], S∈[50, 200], and V∈[50, 150]. A captured image is selected, requiring that the image completely captures the mileage post number plate. The RGB image is converted to an HSV image, and the HSV color component adjuster is applied to adjust the component ranges so that the mileage post number plate is displayed in the image binarization window, and there is no small connection between the characters and the border. The adjusted H, S, and V component ranges are recorded and set as the default values.

[0049] S3-2. Import the image data collected by the left and right cameras into the code to extract the mileage post information, perform preliminary positioning of the mileage post plate area in the HSV color space, extract all contours from the preliminary positioning results, filter the contours according to the aspect ratio of the post plate, and intercept the corresponding area in the image for character segmentation;

[0050] S3-3. Create standard character images of the 10 digits "0" to "9" using the "traffic sign font" and segment the image using the character segmentation code. The minimum matrix image containing the characters is retained as the standard traffic sign digit template.

[0051] S3-4. Identify each segmented mileage post character area one by one. During the identification process, adjust each template and the character image to be matched to the same ratio, calculate the matching score between the mileage post characters and each digital template, take the optimal solution and output it in the order of character segmentation, and rename the original image according to the output template content and store it in the mileage post image information database to complete the recognition and extraction of highway mileage post information.

[0052] Step S4 is specifically as follows:

[0053] S4-1. The sliding window method stipulates that the pixel window is a square of 5 pixels, and the normalized correlation coefficient is used to measure the correlation between the pixel to be matched and the candidate pixel. The normalized correlation coefficient calculation formula is: where p′ u is the optimal pixel matching position; s is the position of all candidate pixels; w and w′ are vectors consisting of the grayscale values ​​of the pixels in the pixel window to be matched and the candidate pixel window respectively;

[0054] S4-2. The formula for calculating depth of field is: Where z is the depth of field; f is the focal length of the camera; B is the baseline length; and d is the parallax.

[0055] Step S5 is specifically as follows:

[0056] S5-1. The formula for converting pixel coordinates into three-dimensional coordinates in the camera coordinate system with the left camera as the origin through geometric relationships is:

[0057]

[0058] Wherein, X, Y, and Z are the three-dimensional coordinates of the target point in the left camera coordinate system; f is the focal length; B is the distance between the two camera coordinate origins, that is, the baseline length; d is the viewing distance of the corresponding pixel point; xl and yl are the X and Y coordinates of the projection point of the left camera image plane in the left camera coordinate system; x0 and y0 are the X and Y coordinates of the origin of the left camera image plane in the left camera coordinate system;

[0059] S5-2. Calculate the vehicle pose using the yaw angle α, pitch angle β, and roll angle γ in the navigation data corresponding to the image acquisition time, expressed as a matrix:

[0060]

[0061]

[0062]

[0063] S5-3, the reprojection matrix calculation formula is:

[0064] M=R(α)*R(β)R(γ)*R1*R2+T1+T2

[0065] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A highway pile number matching method based on binocular stereo vision, characterized in that: The following steps are involved: A navigation system and a binocular camera are installed on a data collection vehicle, the rotation and translation relationship is recorded, and the binocular camera is calibrated and stereo corrected; the data collection vehicle collects road positioning data and image data along the road section to be matched, and the image data includes left camera image data and right camera image data; Processing the road positioning data using a polynomial interpolation method to obtain a driving trajectory of the data collection vehicle; Identify and extract the mileage post number in the image data, name and store the image data according to the mileage post number, and establish a left camera mileage post image database and a right camera mileage post image database respectively; Perform pixel matching on the images of the same mileage post in the left camera mileage post image database and the right camera mileage post image database to obtain disparity and calculate the depth of field, where the pixel position is at the upper left corner of the mileage post number plate; Converting pixel coordinates into three-dimensional coordinates in a camera coordinate system with the left camera as the origin according to the parallax, the depth of field, and the geometric relationship, and obtaining posture information in combination with the driving trajectory; Based on the posture information and the rotation and translation relationship, a reprojection matrix of the camera coordinate system relative to the world coordinate system is obtained when a number of image data are acquired, and the three-dimensional coordinates in the camera coordinate system are converted into the geodetic coordinates corresponding to the milepost based on the reprojection matrix.

2. The highway pile number matching method based on binocular stereo vision according to claim 1 is characterized in that: The calibration process of the binocular camera includes: calibrating each camera in the binocular camera one by one using the Zhang Zhengyou calibration method; setting up and fixing the relatively parallel left and right cameras, capturing a checkerboard image, establishing a coordinate system with the optical center of the left camera as the origin, and combining the captured checkerboard image to obtain the rotation matrix and translation matrix of the right camera relative to the left camera to complete the calibration of the binocular camera.

3. The highway pile number matching method based on binocular stereo vision according to claim 1 is characterized in that: The road positioning data includes: longitude and latitude, elevation, three-way vehicle speed, yaw angle, pitch angle and roll angle.

4. The highway pile number matching method based on binocular stereo vision according to claim 1 is characterized in that: The process of obtaining the driving trajectory includes: The vehicle's motion is decomposed into three directions, including east, north, and sky. Cubic polynomial interpolation is performed on the three directions of motion respectively to fit the continuous motion trajectory between the path points in the three directions. Motion synthesis is then performed to obtain the driving trajectory of the data collection vehicle, where the path points are points that include spatial information and posture information.

5. The highway pile number matching method based on binocular stereo vision according to claim 4 is characterized in that: The process of performing cubic polynomial interpolation on the three directions of motion includes: taking the adjacent path points in the three directions as the starting points and the ending points, respectively, and constructing the trajectory function, velocity function, and acceleration function between the adjacent points; Obtaining constraints of trajectory functions and speed functions in three directions respectively based on the road positioning data obtained by the navigation system at different times; The trajectory function is solved according to the constraints to obtain the unknown coefficients of the trajectory function, and then the position and speed of the vehicle at any time between two adjacent trajectory points in unidirectional motion are obtained, and the motion interpolation results in three directions are obtained respectively.

6. The highway pile number matching method based on binocular stereo vision according to claim 1 is characterized in that: The process of identifying and extracting mileage posts includes: Create an HSV color component adjuster, set the HSV color component range according to the characteristics of highway mileage post number plates, convert the image data into an HSV image, adjust the HSV image using the HSV color component adjuster, perform preliminary positioning of the adjusted image in the HSV color space, extract all contours in the preliminary positioning result, filter the contours according to the aspect ratio of the mileage post number plates, and perform character segmentation on the corresponding areas in the image to obtain character segmentation areas; obtain a standard image of the mileage post number plate, obtain a standard digital template based on the standard image, calculate the matching score of the character segmentation area corresponding to each mileage post and the standard digital template, take the optimal solution and output it in the character segmentation order, rename the image data according to the output template content, and store it in the mileage post image information database.

7. The highway pile number matching method based on binocular stereo vision according to claim 1 is characterized in that: The sliding window method is used for pixel matching. The pixel window is limited to a square with a fixed side length. The normalized correlation coefficient is used to determine the correlation between the pixel to be matched and the candidate pixel.

Images