Automatic path acquisition system and method

The automatic path collection system enhances efficiency by integrating sensors and deep learning for accurate lane line detection and correction, addressing inefficiencies in current manual methods and promoting unmanned operations.

CN119085667APending Publication Date: 2024-12-06JIANGSU JITRI ROAD ENG TECH & EQUIP RES INST CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410998307.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Current methods for collecting vehicle paths in highway maintenance are inefficient, limiting the promotion of unmanned operations, with technicians collecting only 2-3 kilometers per hour using handheld RTK equipment.

Method used

An automatic path collection system utilizing a positioning fixed base station, real-time differential fixed base station, high-precision camera, inertial navigation system, wheel speed meter, vehicle-mounted RTK, and control unit to enhance path collection efficiency, incorporating deep learning and multi-sensor fusion for accurate lane line detection and correction.

Benefits of technology

Improves path collection efficiency, enabling accurate lane line detection and correction, facilitating the promotion of unmanned operations by integrating sensors and deep learning for real-time data processing and error correction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119085667A_ABST
    Figure CN119085667A_ABST
Patent Text Reader

Abstract

The invention discloses an automatic path acquisition system and method. The system comprises a positioning fixed base station, a real-time differential fixed base station, a high-precision camera, an inertial navigation system, a wheel speed meter, a transport vehicle, a vehicle-mounted RTK and a control unit, the method comprises the following steps: initially positioning a vehicle by using a positioning fixed base station and an RTK; the high-precision camera captures an initial image and identifies a reference frame; the inertial navigation system and the wheel speedometer record the current speed of the vehicle, the wheel turning angle and the real-time position and attitude information of the vehicle; the high-precision camera continuously captures lane line images; the real-time differential fixed base station continuously provides position information; the inertial navigation system measures the acceleration and angular velocity of the vehicle; the wheel speedometer measures the rotating speed of wheels; collected information is transmitted to the control unit in real time to detect and recognize lane lines, the position change of a vehicle is calculated through a dead reckoning method, the position information provided by the real-time difference fixed base station is fused with a dead reckoning result, and accumulative errors are corrected. According to the invention, the path acquisition efficiency is improved, and unmanned operation popularization is facilitated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to an automatic path collection system and method, belonging to the technical field of unmanned driving. Background Art

[0002] With the continuous development of unmanned driving technology, its application scenarios are becoming more and more diverse. Highway maintenance operation scenarios are relatively simple, with fixed vehicle driving paths, low driving speeds and less external interference. Therefore, the application of unmanned driving technology in the field of maintenance operations has obvious advantages over traditional fields.

[0003] As a pre-work for highway maintenance, the current method used is for on-site technicians to collect routes by walking and marking points with handheld RTK equipment. This collection method is inefficient and can only collect 2-3 kilometers of routes per hour. For non-continuous construction sites, the collection time will be longer. This inefficient collection greatly limits the promotion of unmanned operations. Summary of the invention

[0004] The purpose of the present invention is to provide a system and method for automatic path collection, which can improve the efficiency of path collection and facilitate the promotion of unmanned operations.

[0005] In order to achieve the above-mentioned object, the present invention provides a path automatic acquisition system, including a positioning fixed base station, a real-time differential fixed base station, a high-precision camera, an inertial navigation system, a wheel speed meter, a transport vehicle, a vehicle-mounted RTK and a control unit;

[0006] The positioning fixed base station is a set fixed position reference point, which is used to calibrate and correct the data using it as a reference point;

[0007] The real-time differential fixed base station uses GPS signals to provide location information and transmits it to the control unit, and eliminates common GPS errors by comparing the data obtained with the data of the positioning fixed base station;

[0008] The high-precision camera is used to capture image information of lane lines and transmit the collected image information to the control unit, which uses image processing technology to detect and identify lane lines;

[0009] The inertial navigation system is used to measure the acceleration and angular velocity of the transport vehicle and transmit the acceleration and angular velocity information to the control unit, which calculates the direction and position changes of the vehicle;

[0010] The wheel speed meter is used to measure the rotation speed of the wheel and transmit the wheel rotation speed information to the control unit, which calculates the travel distance and speed of the vehicle;

[0011] The vehicle-mounted RTK is used to record the driving trajectory of the transport vehicle in real time and transmit the driving trajectory information to the control unit; the control unit converts the driving trajectory data into the coordinate position of the lane line;

[0012] The transport vehicle is used to carry a real-time differential fixed base station, a high-precision camera, an inertial navigation system, a wheel speed meter and a vehicle-mounted RTK to collect data on the road on which it is traveling;

[0013] The control unit is used to receive position information, lane line image information, vehicle acceleration and angular velocity information, wheel speed information and vehicle driving trajectory information transmitted by a positioning fixed base station, a high-precision camera, an inertial navigation system, a wheel speed meter and a vehicle-mounted RTK, and processes the received information to eliminate GPS errors, detect and identify lane lines, calculate vehicle direction and position changes, and calculate vehicle driving distance and speed.

[0014] Furthermore, the real-time differential fixed base station uses GPS signals to provide centimeter-level accuracy position information.

[0015] Furthermore, the high-precision camera has video shooting, image capture and continuous shooting functions; its photo sampling samples cover lane line types such as solid lines, dotted lines, zebra crossings under different lighting conditions, as well as urban roads and highway scenes, forming an original image database.

[0016] Furthermore, the positioning fixed base station is installed at an unobstructed place on the roadside, and its coverage radius is 5 to 8 kilometers.

[0017] A method for automatically collecting paths comprises the following steps:

[0018] Step 1. After the automatic acquisition system is started, the transport vehicle is initially positioned using the fixed positioning base station and RTK. The initial image is captured using a high-precision camera, and the reference frame for visual recognition is set based on the initial image. The inertial navigation system and wheel speedometer begin to record the vehicle's current speed, wheel angle, and real-time position and attitude information for mutual verification with the RTK information.

[0019] Step 2. A high-precision camera continuously captures lane line images; a real-time differential fixed base station continuously provides position information; an inertial navigation system measures the acceleration and angular velocity of the transport vehicle; a wheel speed meter measures the rotation speed of the wheel; the lane line image information, position information, acceleration and angular velocity information and wheel rotation speed information are transmitted to a control unit in real time, the control unit uses image processing technology to detect and identify lane lines, combines the acceleration and angular velocity of the transport vehicle with the wheel rotation speed data, calculates the position change of the vehicle through a dead reckoning method, and fuses the position information provided by the real-time differential fixed base station with the dead reckoning result to correct the accumulated error; at the same time, the control unit records the lane lines and vehicle positions in real time for abnormal condition alarms and subsequent analysis.

[0020] Furthermore, in the step 2, lane line detection and recognition is performed based on a lane line detection model of deep learning, and the specific steps are as follows:

[0021] Step 2-1.1. Use UFLD to achieve high frame rate lane line detection. The calculation formulas of lane line positioning loss function and total detection loss are as follows:

[0022]

[0023] L detection =L cls +α*L sim +γ*L shp ;

[0024] Where M is the number of lane lines, h is the number of image recognition boxes, and L is CE is the cross loss function, P i,j is the probability of the model output unit, T i,j is the output target; α and γ are loss coefficients, both of which are (0,1); L sim is the similarity loss, indicating that the lane lines in adjacent image recognition frames are close to each other; L shp It is the shape loss, which depends on the non-standard state of the lane line. The straight lane line is in the standard state, and the curved lane line is in the non-standard state. The more curved, the more serious the image loss.

[0025] Step 2-1.2: Generate a lane detection and recognition model by constructing an unsupervised adaptive UDA algorithm for images. The specific steps are as follows:

[0026] a. Use a high-precision camera to take photos of lane lines of different forms and construct an original image database;

[0027] b. Use convolutional neural networks to learn and extract useful features from the original image database. Specifically, the convolution layer extracts local features of the image. Each convolution layer contains a set of convolution kernels, which are used to slide on the input feature map and perform convolution calculations to generate new feature maps. The convolution calculation formula is as follows:

[0028]

[0029] In the formula, Represents the value of the kth channel at position (i, j) of the feature map obtained after the lth convolution layer;

[0030] is the weight of the kth channel of the lth convolution kernel at position (p, q); is the activation value of the kth channel of the output feature map of the (l-1)th layer at position (i+p,j+q); is the bias term of the kth channel of the lth layer convolution kernel; P and Q are the height and width of the convolution kernel;

[0031] c. The feature map processed in step b is passed through the pooling layer to reduce the dimension while retaining important information. The pooling formula is as follows:

[0032] Y ij =max 0<x<W,0<y<H {X i×S+x,j×S+y};

[0033] Where Y i,j Represents the value of the feature map at position (i, j) after pooling; X i×S+x,j×S+y Represents the value of the input feature map within the pooling window, where S is the pooling step size, W and H are the width and height of the pooling window;

[0034] d. Regularize the pooled feature map. The formula is as follows:

[0035]

[0036] Where J is the regularized loss function; J0 is the original loss function; λ is the regularization coefficient; ∑ w∈W w 2 is the set of all weights in the model; m is the number of training samples; W is the weight in the model, that is, the parameter to be optimized; the training and optimization refers to the use of BP algorithm to calculate the loss function to optimize the gradient to speed up the convergence of the model. The calculation process is as follows:

[0037]

[0038] In the formula, y i is the label of the i-th sample; z iis the corresponding predicted value; W j represents weight; x ij is the jth eigenvalue of the i-th sample; b is the bias term; is the updated weight; b + is the updated bias; α is the learning rate;

[0039] Step 2-1.3, calculate the relative distance of the lane edge and record the key frame pose in the absolute coordinate global coordinate system. The steps are as follows:

[0040] a. Selecting specific frames with clear images from images or video frames taken by a high-precision camera as key frames to calculate the changes between adjacent frames;

[0041] b. The key frame selected in step a is converted from the local frame to the global coordinate. The specific conversion process is as follows: the key frame image is optimized by the factor graph and divided into three stages: initialization, external calibration and positioning. Among them, the initialization performs initial external parameter estimation, that is, a clear picture is generated after a set time interval based on the previous frame, and there is enough movement with the previous node, that is, after there is obvious color difference or feature variable with the previous picture, a new key frame is selected, and the pair of key frame postures are inserted into the factor graph for time alignment through spherical linear attitude interpolation; after collecting N nodes in each frame, it is determined that the initialization is successful; the value of N is the distance traveled by the vehicle per second, the unit is meter / S, and it is rounded up; among them, the obvious color difference is distinguished by comparing the color histogram, and the process is as follows:

[0042] import cv2

[0043] import numpy as np

[0044] #Read the image

[0045] image1=cv2.imread('sunset1.jpg')

[0046] image2=cv2.imread('sunset2.jpg')

[0047] # Calculate the histogram for each channel

[0048] histSize = 256 # number of bins

[0049] histRange = (0,256) # range

[0050] hist_full=cv2.calcHist([image1],[0,1,2],None,[histSize,histSize,histSize],[histRange,histRange,histRange])

[0051] hist_flat=hist_full.flatten()

[0052] #Do the same for the second image

[0053] cv2.normalize(hist_flat,hist_flat,alpha=0,beta=1,norm_type=cv2.NORM_MINMAX)

[0054] similarity=cv2.compareHist(hist_flat1,hist_flat2,cv2.HISTCMP_CORREL)

[0055] #The return value is between -1 and 1, where 1 means completely similar;

[0056] External calibration is to construct a ternary edge, constrain the key frame posture from two frames of images and external parameters respectively, so as to accurately solve the coordinate transformation relationship. The steps of constraining the key frame posture are as follows:

[0057] ① Extract feature points from the image and use the visual odometer method to estimate the relative motion of the high-precision camera based on the matching of feature points between consecutive frames;

[0058] ② Determine the picture with sufficient motion relative to the previous node and establish a new key frame;

[0059] ③ Temporally align the pose of the new keyframe with the pose of the previous keyframe through pose interpolation;

[0060] ④ Repeat steps ① to ③. After collecting a set number of key frames, use the factor graph to optimize the poses and external parameters of all key frames.

[0061] The positioning, i.e. coordinate transformation process, is as follows:

[0062] ① Constructing ternary edge constraints in the factor graph, which are used to encode the geometric relationship between keyframes and between keyframes and extrinsic parameters, which are derived from the relative pose between two frames and GPS and IMU data information;

[0063] ② Use the Levenberg-Marquardt algorithm to minimize the errors of all edges in the factor graph, accurately solve the coordinate transformation relationship, and then transform the relative pose obtained in the global coordinate system into the global coordinate; the calculation method of the global coordinate is:

[0064] The distance information of the feature point relative to the high-precision camera coordinate system is obtained by using a high-precision camera, which is the relative coordinate (x1, y1). From the calibration parameters of the antenna and the high-precision camera, it is known that the relative coordinates from the feature point to the antenna are (x1+x2, y1); the distance from the feature point to the antenna is L2, then:

[0065]

[0066] According to the positional relationship between the positioning antenna and the high-precision camera, the angle σ between the feature point and the line connecting the positioning antenna relative to the true north direction is obtained. From the trigonometric function, it can be known that the distances of the feature points relative to the positioning antenna are:

[0067] x i =L2sinσ

[0068] y i =L2cosσ;

[0069] Step 2-1.4, calculate the vehicle driving coordinates, the process is:

[0070] Before path collection, a positioning fixed base station is fixed at a suitable position on the roadside to give a relative coordinate origin. The purpose of the base station is to eliminate the clock error and multipath effect of the satellite positioning signal. The fixed base station is regarded as the origin of the coordinate system; a differential fixed base station is fixedly installed at a specified position on the transport vehicle. When the transport vehicle is driving, the RTK, fixed base station and satellite communicate with each other to obtain the trajectory coordinates of the RTK as the vehicle travels in real time, and the coordinates are corrected in real time through the inertial navigation system and wheel speedometer; when the directional antenna loses signal for a short time, the inertial navigation system is relied on to provide continuous navigation. Since the inertial navigation system itself has cumulative errors, the inertial navigation system and the positioning antenna correct each other during the navigation process. When the deviation between the inertial navigation information and the navigation information is greater than 5%, an alarm is issued;

[0071] Step 2-1.5, calculate the lane line coordinates. The process is as follows:

[0072] According to the RTK coordinate trajectory obtained in real time, combined with the relative distance information of the lane line from the vehicle identified by the camera, the spatial trigonometric function relationship in Step 2-1.3 is used to fused and analyze the camera with RTK, inertial navigation system, and wheel speed meter by jointly calibrating multiple sensors at the same time. Combined with the data association algorithm, the position information of the lane line is output through the data relationship of the same time frame, and the RTK real-time trajectory is converted into the coordinate trajectory of the lane line, and this trajectory is designated as the task path; the specific formula is as follows:

[0073]

[0074] In the formula, is the state estimate of the transport vehicle at time (i+1);

[0075] is the state estimate of the transport vehicle at time i;

[0076] is the average speed of the transport vehicle in the time period [i, i+1], δ t is the time interval.

[0077] Furthermore, in the step 2, the method for correcting the cumulative error is: the inertial navigation system, the wheel speed meter and the RTK equipment perform point detection and correction on the vehicle driving trajectory through a multi-sensor fusion filtering algorithm. When the inertial navigation system, the wheel speed meter and the RTK acquisition path deviate, the error parameters are eliminated through integral calculation to correct the final path trajectory parameters, specifically:

[0078] Step 2-2.1, use the system timestamp to ensure the time synchronization of sensor data;

[0079] Step 2-2.2, compare the trajectory data of different sensors, and mark them as deviations if they exceed the set threshold; the system error is usually set within 1%;

[0080] Step 2-2.3, based on the vehicle's kinematic model, calculate whether the vehicle's acceleration and steering angle at the next moment are in line with expectations; the vehicle's kinematic model uses point F as the control reference point, L as the vehicle's axle spacing, R as the turning radius of the front wheels, α and β as the turning angles of the front and rear wheels, respectively, and V F is the speed of the front wheel, γ is the angle between the vehicle body and the X-axis, also known as the heading angle, and the formula is as follows:

[0081]

[0082] The calculation process of whether the vehicle's acceleration and steering angle at the next moment meet expectations is:

[0083] During vehicle driving, after obtaining the inertial navigation system and wheel predicted scores, the consistency of the predicted position variance is adjusted and dynamically adjusted, and the noise re-estimation is applied to the extraction and recognition of image features. The formula is as follows:

[0084]

[0085] In the formula, ||γ p -H p X|| 2 is the error between the prior information and the state variable; is the error between the inertial measurement unit and the state variable; is the error between the wheel and the state variable; is the error between the reprojection error between adjacent frames and the state variable; X is the state variable, and the four residuals correspond to the prior information of marginalization, inertial navigation system measurement, wheel measurement and reprojection error between adjacent frames, respectively.

[0086] Furthermore, in the step 2, the lane line images continuously captured by the high-precision camera are used by the control unit to establish a mathematical model by combining an image recognition algorithm and deep learning, thereby increasing the recognition rate of the lane line and the distance measurement accuracy; the mathematical model construction process combining the image recognition algorithm and deep learning is as follows:

[0087] Step 2-3.1, pre-process the images continuously captured by the high-precision camera, including image cleaning, cropping, and enhancement, to improve image quality and reduce noise interference;

[0088] Step 2-3.2: Use image recognition algorithms to extract lane line features in the image, including color, shape, and texture; these features will serve as input for subsequent deep learning models;

[0089] Step 2-3.3, use the YOLO algorithm model to learn and train the image, and evaluate the trained model, including calculating the accuracy and recall rate; optimize the model according to the evaluation results, including adjusting the network structure and increasing the training data;

[0090] Step 2-3.4, use the weighted cross entropy function to calculate the lane line recognition loss function, the formula is as follows:

[0091]

[0092] In the formula, C represents the number of categories, i and j represent the i-th sample and j-th category respectively; y is the true value of the sample, y ij Indicates the true value of the sample belonging to the jth category. After Onehot encoding, y ij =0 or y ij =1; pij represents the output probability of the predicted value of the i-th sample in the j-th category; w j Represents the weight of the j-th classification, which is expressed as follows:

[0093]

[0094] Among them, med_pixel_j represents the median value of the j-class pixel points in the image; sum_pixel_j represents the sum of the j-class pixel points in the image; sum_pixel_all represents the sum of all image sample pixel points.

[0095] Furthermore, in order to improve the calculation efficiency of the deviation of the acquisition path, the Jacobian matrix relative to the translation variational solution is calculated according to the obtained attitude angle as follows:

[0096]

[0097] Where θ represents the rotation angle, a is the unit vector rotation axis, (*)^ is a skew-symmetric matrix, which converts the linear velocity and acceleration of the wheel into the IMU frame and is expressed as:

[0098]

[0099] w oI It is the angular velocity generated by the rigid connection. By multiplying the Jacobian matrix, the velocity and the most recent time interval, the translation required by the system is obtained. The translation required by the system means that the system will obtain two coordinates when collecting coordinates. One is the coordinate point generated by the movement of the positioning antenna itself, and the other is the coordinate point of the lane line feature point relative to the camera after the camera takes pictures. Finally, the coordinate point obtained by the camera is converted into the coordinate information of the positioning directional antenna. Here, not only the relative position of the camera and the positioning directional antenna should be considered, but also the time difference when the two collect information and the relative coordinate change between the two within the time difference should be considered.

[0100] The present invention sets a positioning fixed base station, a real-time differential fixed base station, a high-precision camera, an inertial navigation system (IMU), a wheel speed meter, a transport vehicle, a vehicle-mounted RTK and a control unit. After the automatic acquisition system is started, the positioning fixed base station and RTK are used to perform initial positioning on the transport vehicle; the high-precision camera is used to capture the initial image, and a reference frame for visual recognition is set according to the initial image; the inertial navigation system and the wheel speed meter start to record the current speed, wheel angle and real-time position and posture information of the vehicle for mutual verification with the RTK information; the high-precision camera continuously captures lane line images; the real-time differential fixed base station continuously provides position information ; The inertial navigation system measures the acceleration and angular velocity of the transport vehicle; the wheel speed meter measures the rotation speed of the wheel; the lane line image information, position information, acceleration and angular velocity information and wheel rotation speed information are transmitted to the control unit in real time, and the control unit uses image processing technology to detect and identify the lane line, combines the acceleration and angular velocity of the transport vehicle with the wheel rotation speed data, calculates the position change of the vehicle through the dead reckoning method, and integrates the position information provided by the real-time differential fixed base station with the dead reckoning result to correct the accumulated error; at the same time, the control unit records the lane line and vehicle position in real time for abnormal condition alarm and subsequent analysis. The present invention improves the efficiency of path collection and is conducive to the promotion of unmanned operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] Figure 1 It is a schematic diagram of the distance of the characteristic point relative to the camera coordinate system obtained by the present invention using the camera;

[0102] Figure 2 It is a schematic diagram of the relationship of coordinate transformation of the present invention;

[0103] Figure 3 It is a calculation flow chart of the fusion algorithm of the present invention. DETAILED DESCRIPTION

[0104] The present invention will be further described below in conjunction with the accompanying drawings.

[0105] The present invention combines the initialization values ​​of the system under static and dynamic working conditions, estimates the bias of the accelerometer and the gyroscope according to the static IMU measurement value as the reference value of dynamic initialization, optimizes the bias of the accelerometer and the gyroscope during the dynamic initialization process, estimates the initial state variables of the system, and couples and analyzes the above variables using an optimization algorithm to obtain the real-time position and posture of the current vehicle, integrates the linear velocity of the vehicle, and verifies the accuracy of the vehicle satellite data in real time;

[0106] like Figure 1 and Figure 2As shown, in the system of the present invention, the key frame pose in the local coordinate system and the key frame pose in the ENU coordinate system as the absolute coordinate are maintained. In order to realize the relative pose constraint based on VIWO or IWO, the pose needs to be converted from the local frame to the ENU coordinate. In other words, the corresponding external parameters need to be estimated. Theoretically, the coordinate transformation relationship of the factor graph is unchanged in the absence of a true value reference, and the factor graph optimization can be divided into three stages: initialization, external calibration, and positioning.

[0107] The initialization process is responsible for the initial external parameter estimation. Once enough motion is completed based on the previous node, a new keyframe is selected and a pair of keyframe poses are inserted into the factor graph for temporal alignment through pose interpolation. After collecting 15 nodes per frame, the initialization is determined to be successful.

[0108] In the second stage, the three-element edge is constructed to constrain the key frame posture from the two frames and external parameters respectively, which is used to accurately solve the coordinate transformation relationship. The relative posture obtained in the local coordinate system is converted to the global coordinate system (ENU), which mainly provides angle constraints.

[0109] After calibrating the extrinsic parameters between the local frame and the global coordinates (ENU), we will keep the extrinsic parameters unchanged and remove them from the variables that need to be adjusted during the optimization process.

[0110] Lane line recognition images include but are not limited to the following types: overtaking lanes, fast lanes and driving lanes on the main road; interchange ramps, acceleration ramps, deceleration ramps, approaches, distribution ramps and turning ramps on the road; emergency lanes, U-turn lanes, climbing lanes, escape lanes and cooling lanes in the auxiliary lanes. At the same time, considering that the construction area is mainly for roads with a long service life, lane line recognition also includes damaged lanes.

[0111] Embodiment: (1) The system is started, and the vehicle travels along the path to be collected, with the speed controlled at 5-10 km / h; the image obtained by the camera is preprocessed, including video frame segmentation and video annotation; the collected video information is segmented at intervals of 30 frames. The number of frame segmentations set here should be adjusted according to the performance of your own equipment, and does not mean that it must be segmented according to 30 frames; Gaussian filtering is used for noise reduction processing, which is achieved by replacing the value of each pixel with the weighted average of adjacent pixels. The weight is given by the Gaussian function, and the corresponding mathematical expression is:

[0112] Output(x,y)=∑ (s,t)∈Neighborhood(x,y) G(sx,ty)·Input(s,t);

[0113] Among them, Neighborhood(x,y) represents the neighborhood of pixel (x,y), and G(x,y) is a two-dimensional Gaussian function:

[0114]

[0115] Output(x,y) represents the output pixel value after Gaussian filtering at the coordinate (x,y); Input(s,t): represents the pixel value of the original image at (s,t); (s,t)∈Neighborhood(x,y): represents that the coordinate (s,t) belongs to a neighborhood of the coordinate (x,y);

[0116] G(sx,ty): Gaussian function used to calculate weights. This function assigns weights based on the distance between the pixel point (s,t) and the center pixel point (x,y). The closer the distance, the greater the weight, and the farther the distance, the smaller the weight. sx and ty represent the horizontal and vertical distances respectively; σ: standard deviation of the Gaussian function, which determines the width of the Gaussian function;

[0117] (2) The histogram equalization method is used to enhance the contrast of the data. The relevant mathematical expression is as follows:

[0118]

[0119] Where s is the output grayscale, L is the number of grayscale levels, which is generally 256, and P r (k) is the probability of pixels with gray level k in the input image. The following example illustrates the steps of implementing the histogram equalization method:

[0120] Assume a simple 4x4 grayscale image with pixel values ​​as shown in Table 1:

[0121] Table 1 Pixel values

[0122] 50 60 70 80 60 70 80 90 70 80 90 100 80 90 100 110

[0123] The gray level histogram of the image is shown in Table 2:

[0124] Table 2 Gray level histogram

[0125]

[0126]

[0127] The calculated cumulative distribution function (CDF) is shown in Table 3:

[0128] Table 3 Cumulative distribution function

[0129] Grayscale CDF 50 0.0625 60 0.1875 70 0.375 80 0.625 90 0.8125 100 0.9375 110 1

[0130] Map pixel values ​​and map the CDF values ​​to the range of 0 to 255 (the number of gray levels of L is 256). The value of each CDF after mapping is (CDF*255 rounded up) as shown in Table 4:

[0131] Table 4 Mapping pixel values

[0132] Grayscale CDF 50 16 60 48 70 96 80 159 90 207 100 239 110 255

[0133] The new image grayscale values ​​are shown in Table 5:

[0134] Table 5 Image grayscale values

[0135] 16 48 96 159 48 96 159 207 96 159 207 239 159 207 239 255

[0136] (3) Use the UFLD algorithm to process the enhanced image data to obtain specific lane line information: The ResNet34 network structure model is used here, and the calculation formulas for the lane line positioning loss function and the total detection loss are as follows:

[0137]

[0138] L detection =L cls +α*L sim +γ*L shp

[0139] Where M is the number of lane lines, h is the number of image recognition boxes, and L is CE is the cross loss function, P i,j is the probability of the model output unit, T i,j is the output target; α and γ are loss coefficients, both of which are (0,1); L sim is the similarity loss, indicating that the lane lines in adjacent image recognition frames are close to each other; L shp It is the shape loss, which depends on the non-standard state of the lane line; the lane line in the straight state is the standard state, and the lane line in the curved state is the non-standard state. The more curved, the more serious the image loss. The image data processing process is shown in Table 6:

[0140] Table 6 Image data processing

[0141]

[0142]

[0143] (4) The residual structure is shown in Table 7:

[0144] Table 7 Residual structure

[0145]

[0146]

[0147] (5) The training model is shown in Table 8:

[0148] Table 8 Training model

[0149]

[0150] (6) Save the training results as a .pt file and transfer the file to the vehicle path acquisition system;

[0151] (7) Start the path collection system, let the collection vehicle drive along the lane line, and save the collected path curve in a specified format. The specified format for this project is .project format;

[0152] (8) Import the file into the unmanned driving system and assign tasks to each vehicle. Figure 3 The control flow shown is for unmanned driving.

Claims

1. A route automatic acquisition system, comprising a transport vehicle, a high-precision camera, an inertial navigation system, a vehicle-mounted RTK and a wheel speed meter, characterized in that: It also includes a positioning fixed base station, a real-time differential fixed base station and a control unit; The transport vehicle is used to carry a real-time differential fixed base station, a high-precision camera, an inertial navigation system, a wheel speed meter and a vehicle-mounted RTK to collect data on the road on which it is traveling; The high-precision camera is used to capture image information of lane lines and transmit the collected image information to the control unit, which uses image processing technology to detect and identify lane lines; The inertial navigation system is used to measure the acceleration and angular velocity of the transport vehicle and transmit the acceleration and angular velocity information to the control unit, which calculates the direction and position changes of the vehicle; The vehicle-mounted RTK is used to record the driving trajectory of the transport vehicle in real time and transmit the driving trajectory information to the control unit; the control unit converts the driving trajectory data into the coordinate position of the lane line; The wheel speed meter is used to measure the rotation speed of the wheel and transmit the wheel rotation speed information to the control unit, which calculates the travel distance and speed of the vehicle; The positioning fixed base station is a set fixed position reference point, which is used to calibrate and correct the data using it as a reference point; The real-time differential fixed base station uses GPS signals to provide location information and transmits it to the control unit, and eliminates common GPS errors by comparing the data obtained with the data of the positioning fixed base station; The control unit is used to receive position information, lane line image information, vehicle acceleration and angular velocity information, wheel speed information and vehicle driving trajectory information transmitted by a positioning fixed base station, a high-precision camera, an inertial navigation system, a wheel speed meter and a vehicle-mounted RTK, and processes the received information to eliminate GPS errors, detect and identify lane lines, calculate vehicle direction and position changes, and calculate vehicle driving distance and speed.

2. The automatic path collection system according to claim 1, characterized in that: The real-time differential fixed base station uses GPS signals to provide centimeter-level accuracy position information.

3. The automatic path collection system according to claim 1 or 2, characterized in that: The high-precision camera has video shooting, image capture and continuous shooting functions; its photo sampling samples cover lane line types such as solid lines, dotted lines, zebra crossings under different lighting conditions, as well as urban roads and highway scenes, forming an original image database.

4. The automatic path collection system according to claim 1, characterized in that: The positioning fixed base station is installed at an unobstructed place on the roadside, and its coverage radius is 5 to 8 kilometers.

5. A method for automatic path collection, characterized in that: The steps include: Step 1. After the automatic acquisition system is started, the transport vehicle is initially positioned using the fixed positioning base station and RTK. The initial image is captured using a high-precision camera, and the reference frame for visual recognition is set based on the initial image. The inertial navigation system and wheel speedometer begin to record the vehicle's current speed, wheel angle, and real-time position and attitude information for mutual verification with the RTK information. Step 2: The high-precision camera continuously captures lane line images; the real-time differential fixed base station continuously provides location information; the inertial navigation system measures the acceleration and angular velocity of the transport vehicle; The wheel speed meter measures the rotation speed of the wheel; the lane line image information, position information, acceleration and angular velocity information and wheel rotation speed information are transmitted to the control unit in real time, and the control unit uses image processing technology to detect and identify the lane line, combines the acceleration and angular velocity of the transport vehicle with the wheel rotation speed data, calculates the position change of the vehicle through the dead reckoning method, and fuses the position information provided by the real-time differential fixed base station with the dead reckoning result to correct the accumulated error; at the same time, the control unit records the lane line and vehicle position in real time for abnormal condition alarm and subsequent analysis.

6. The method for automatically collecting paths according to claim 5, characterized in that: In step 2, lane line detection and recognition is performed based on a lane line detection model based on deep learning, and the specific steps are as follows: Step 2-1.

1. Use UFLD to achieve high frame rate lane line detection. The calculation formulas of lane line positioning loss function and total detection loss are as follows: L detection =L cls +a*L sim +γ*L shp ; Where M is the number of lane lines, h is the number of image recognition boxes, and L is CE is the cross loss function, P i,j is the probability of the model output unit, T i,j is the output target; α and γ are loss coefficients, both of which are (0,1); L sim is the similarity loss, indicating that the lane lines in adjacent image recognition frames are close to each other; L shp is the shape loss, which depends on the non-standard state of the lane line; Step 2-1.2: Generate a lane detection and recognition model by constructing an unsupervised adaptive UDA algorithm for images. The specific steps are as follows: a. Use a high-precision camera to take photos of lane lines of different forms and construct an original image database; b. Use convolutional neural networks to learn and extract useful features from the original image database. Specifically, the convolution layer extracts local features of the image. Each convolution layer contains a set of convolution kernels, which are used to slide on the input feature map and perform convolution calculations to generate new feature maps. The convolution calculation formula is as follows: In the formula, Represents the value of the kth channel at position (i, j) of the feature map obtained after the lth convolution layer; is the weight of the kth channel of the lth convolution kernel at position (p, q); is the activation value of the kth channel at position (i+p, j+q) of the output feature map of the (l-1)th layer; is the bias term of the kth channel of the ,th layer convolution kernel; P and Q are the height and width of the convolution kernel; c. The feature map processed in step b is passed through the pooling layer to reduce the dimension while retaining important information. The pooling formula is as follows: AND i,j =max 0<x<W,0<y<H {X i×S+x,j×S+y }; Where Y i,j Represents the value of the feature map at position (i, j) after pooling; X i×S+x,j×S+y Represents the value of the input feature map within the pooling window, where S is the pooling step size, W and H are the width and height of the pooling window; d. Regularize the pooled feature map. The formula is as follows: Where J is the regularized loss function; J0 is the original loss function; λ is the regularization coefficient; ∑ w∈W w 2 is the set of all weights in the model; m is the number of training samples; W is the weight in the model, that is, the parameter to be optimized; the training and optimization refers to the use of BP algorithm to calculate the loss function to optimize the gradient to speed up the convergence of the model. The calculation process is as follows: In the formula, y i is the label of the i-th sample; z i is the corresponding predicted value; W j represents weight; x ij is the jth eigenvalue of the i-th sample; b is the bias term; is the updated weight; b + is the updated bias; α is the learning rate; Step 2-1.3, calculate the relative distance of the lane edge and record the key frame pose in the absolute coordinate global coordinate system. The steps are as follows: a. Selecting specific frames with clear images from images or video frames taken by a high-precision camera as key frames to calculate the changes between adjacent frames; b. The key frame selected in step a is converted from the local frame to the global coordinate. The specific conversion process is as follows: the key frame image is optimized through the factor graph and divided into three stages: initialization, external calibration and positioning. Among them, the initialization performs initial external parameter estimation, that is, a clear picture is generated after a set time interval based on the previous frame, and there is enough movement with the previous node, that is, after there is obvious color difference or feature variable with the previous picture, a new key frame is selected, and the pair of key frame postures are inserted into the factor graph for time alignment through spherical linear attitude interpolation; after collecting N nodes in each frame, it is determined that the initialization is successful; the value of N is the distance traveled by the vehicle per second, the unit is meter / S, and it is rounded up; External calibration is to construct a ternary edge, constrain the key frame posture from two frames of images and external parameters respectively, so as to accurately solve the coordinate transformation relationship. The steps of constraining the key frame posture are as follows: ① Extract feature points from the image and use the visual odometer method to estimate the relative motion of the high-precision camera based on the matching of feature points between consecutive frames; ② Determine the picture with sufficient motion relative to the previous node and establish a new key frame; ③ Temporally align the pose of the new keyframe with the pose of the previous keyframe through pose interpolation; ④ Repeat steps ① to ③. After collecting a set number of key frames, use the factor graph to optimize the poses and external parameters of all key frames. The positioning, i.e. coordinate transformation process, is as follows: ① Constructing ternary edge constraints in the factor graph, which are used to encode the geometric relationship between keyframes and between keyframes and extrinsic parameters, which are derived from the relative pose between two frames and GPS and IMU data information; ② Use the Levenberg-Marquardt algorithm to minimize the errors of all edges in the factor graph, accurately solve the coordinate transformation relationship, and then transform the relative pose obtained in the global coordinate system into the global coordinate; the calculation method of the global coordinate is: The distance information of the feature point relative to the high-precision camera coordinate system is obtained by using a high-precision camera, which is the relative coordinate (x1, y1). According to the calibration parameters of the antenna and the high-precision camera, the relative coordinates of the feature point to the antenna are (x1+x2, y1); the distance from the feature point to the antenna is L2, then: According to the positional relationship between the positioning antenna and the high-precision camera, the angle σ between the feature point and the line connecting the positioning antenna relative to the true north direction is obtained. From the trigonometric function, it can be known that the distances of the feature points relative to the positioning antenna are: x i =L1sinσ <h2 style=";text-align:left;direction:ltr">y<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> (L2cosσ) Step 2-1.4, calculate the vehicle driving coordinates, the process is: Before path collection, a positioning fixed base station is fixed at a suitable position on the roadside to give the relative coordinate origin, and the fixed base station is regarded as the origin of the coordinate system; a differential fixed base station is fixedly installed at a specified position on the transport vehicle. When the transport vehicle is driving, the RTK, fixed base station and satellite communicate to obtain the track coordinates of the RTK along with the vehicle in real time, and the coordinates are corrected in real time through the inertial navigation system and wheel speed meter; when the directional antenna loses signal for a short time, continuous navigation is provided by relying on the inertial navigation. The inertial navigation and positioning antenna correct each other during the navigation process, and an alarm is issued when the deviation between the inertial navigation information and the navigation information is greater than 5%; Step 2-1.5, calculate the lane line coordinates. The process is as follows: According to the RTK coordinate trajectory obtained in real time, combined with the relative distance information of the lane line from the vehicle identified by the camera, the spatial trigonometric function relationship in Step 2-1.3 is used to fused and analyze the camera with RTK, inertial navigation system, and wheel speed meter by jointly calibrating multiple sensors at the same time. Combined with the data association algorithm, the position information of the lane line is output through the data relationship of the same time frame, and the RTK real-time trajectory is converted into the coordinate trajectory of the lane line, and this trajectory is designated as the task path; the specific formula is as follows: In the formula, is the state estimate of the transport vehicle at time (i+1); is the state estimate of the transport vehicle at time i; is the average speed of the transport vehicle in the time period [i, i+1], δ t is the time interval.

7. The method for automatically collecting paths according to claim 5, characterized in that: In the step 2, the method for correcting the accumulated error is: the inertial navigation system, the wheel speed meter and the RTK equipment detect and correct the vehicle's driving trajectory through a multi-sensor fusion filtering algorithm. When the inertial navigation system, the wheel speed meter and the RTK acquisition path deviate, the error parameters are eliminated through integral calculation to correct the final path trajectory parameters, specifically: Step 2-2.1, use the system timestamp to ensure the time synchronization of sensor data; Step 2-2.2, compare the trajectory data of different sensors, and mark them as deviations if they exceed the set threshold; Step 2-2.3, based on the vehicle's kinematic model, calculate whether the vehicle's acceleration and steering angle at the next moment are in line with expectations; the vehicle's kinematic model uses point F as the control reference point, L as the vehicle's axle spacing, R as the turning radius of the front wheels, α and β as the turning angles of the front and rear wheels, respectively, and V F is the speed of the front wheel, γ is the angle between the vehicle body and the X-axis, also known as the heading angle, and the formula is as follows: The calculation process of whether the vehicle's acceleration and steering angle at the next moment meet expectations is: During vehicle driving, after obtaining the inertial navigation system and wheel predicted scores, the consistency of the predicted position variance is adjusted and dynamically adjusted, and the noise re-estimation is applied to the extraction and recognition of image features. The formula is as follows: In the formula, ||γ p -H p X|| 2 is the error between the prior information and the state variable; is the error between the inertial measurement unit and the state variable; is the error between the wheel and the state variable; is the error between the reprojection error between adjacent frames and the state variable; X is the state variable, and the four residuals correspond to the prior information of marginalization, inertial navigation system measurement, wheel measurement and reprojection error between adjacent frames, respectively.

8. The method for automatically collecting paths according to claim 5, characterized in that: In the step 2, the lane line images continuously captured by the high-precision camera are used by the control unit to establish a mathematical model by combining an image recognition algorithm and deep learning, thereby increasing the recognition rate of the lane line and the distance measurement accuracy; the mathematical model construction process combining the image recognition algorithm and deep learning is as follows: Step 2-3.1, pre-process the images continuously captured by the high-precision camera, including image cleaning, cropping, and enhancement; Step 2-3.2, use image recognition algorithm to extract lane line features in the image, including color, shape, and texture; Step 2-3.3, use the YOLO algorithm model to learn and train the image, and evaluate the trained model, including calculating the accuracy and recall rate; optimize the model according to the evaluation results, including adjusting the network structure and increasing the training data; Step 2-3.4, use the weighted cross entropy function to calculate the lane line recognition loss function, the formula is as follows: In the formula, C represents the number of categories, i and j represent the i-th sample and j-th category respectively; y is the true value of the sample, y ij Indicates the true value of the sample belonging to the jth category. After Onehot encoding, y ij =0 or y ij =1; p ij represents the output probability of the predicted value of the i-th sample in the j-th category; w j Represents the weight of the j-th classification, which is expressed as follows: Among them, med_pixel_j represents the median value of the j-class pixel points in the image; sum_pixel_j represents the sum of the j-class pixel points in the image; sum_pixel_all represents the sum of all image sample pixel points.

9. The method for automatically collecting paths according to claim 7, characterized in that: In order to improve the calculation efficiency of the deviation of the acquisition path, the Jacobian matrix relative to the translation variation solution is calculated according to the obtained attitude angle as follows: Where θ represents the rotation angle, a is the unit vector rotation axis, (*)^ is a skew-symmetric matrix, which converts the linear velocity and acceleration of the wheel into the IMU frame and is expressed as: w oI is the angular velocity generated by the rigid connection. By multiplying the Jacobian matrix, the velocity and the nearest time interval, the required translation of the system is obtained.

Citation Information

Cited By

  • Monitoring method, device and equipment for preventing vehicle-mounted positioning equipment on vehicle from being dismounted

    CN119901315A

  • Map engine adaptive conversion method and system for microscopic traffic simulation data

    CN122220413A