Navigation method based on rtk laser vision fusion

Abstract

The invention discloses a navigation method based on rtk laser vision fusion, and belongs to the field of map navigation, and the method comprises the following steps: S1, data acquisition, RTK data acquisition, and installation of RTK positioning equipment including a base station and a mobile station; the base station is arranged at a fixed point of a known accurate position and continuously transmits a signal containing position information; the mobile station is installed on a navigation carrier, receives base station signals and satellite signals and determines high-precision position coordinates and time information of the carrier, and the advantages of all the sensors can be brought into full play by fusing data of the RTK, the laser radar and the visual sensor, and the limitation of a single sensor is made up. High-precision absolute position information is provided through RTK, the laser radar accurately senses the three-dimensional shape and distance of an object, and the visual sensor identifies specific marks and textures. The fusion of the three can improve the estimation precision of the navigation system on the position, speed and direction, and reduce the positioning error.

Description

Technical Field

[0001] The present invention relates to the field of map navigation technology, and in particular to a navigation method based on RTK laser vision fusion. Background Art

[0002] With the continuous development of navigation technology, a single sensor is often unable to meet the needs of high-precision and high-reliability navigation in complex environments.

[0003] For example, RTK positioning can provide high-precision position information under the open sky, but the accuracy may be reduced due to signal obstruction in urban canyons, indoor environments, etc.; LiDAR can accurately perceive the three-dimensional structure of the surrounding environment, but it may have difficulty detecting long-distance or transparent objects;

[0004] Visual sensors can obtain rich texture and color information, but are greatly affected by factors such as lighting and weather. Therefore, integrating the advantages of multiple sensors has become an inevitable trend to improve navigation performance.

[0005] Modern navigation faces an increasingly complex environment, including various terrains, dynamic obstacles, changing weather and lighting conditions, etc. In such an environment, a navigation method is needed that can fully and accurately perceive the environment and perform efficient path planning and precise control. For example, in urban traffic, vehicles need to avoid pedestrians and other vehicles while following traffic rules; in the field of industrial automation, robots need to move and perform tasks accurately in complex factory environments.

[0006] Therefore, a navigation method based on RTK laser vision fusion is needed to solve the above problems. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides a navigation method based on RTK laser vision fusion, which solves the problems raised in the above background technology.

[0008] Technical solution: To solve the above technical problems, according to one aspect of the present invention, more specifically, a navigation method based on RTK laser vision fusion, comprising the following steps: S1, data acquisition

[0009] RTK data collection

[0010] Install RTK positioning equipment, including a base station and a mobile station; the base station is set at a fixed point with a known precise position and continuously transmits signals containing position information; the mobile station is installed on the navigation carrier, receives the base station signal and satellite signals, and determines the carrier's high-precision position coordinates (longitude, latitude, altitude) and time information;

[0011] Perform a preliminary screening of the RTK data to remove obvious outliers (the positioning data at a certain moment is too different from the data at the previous and next moments, which can be temporarily identified as an outlier and marked for further processing);

[0012] LiDAR data collection

[0013] Start the laser radar to scan the surrounding environment and obtain three-dimensional point cloud data; the point cloud data contains the distance and angle information of objects in the environment;

[0014] De-noising the point cloud data: Statistical filtering is used to remove noise points caused by environmental interference or laser radar errors. Statistical filtering calculates the average distance from each point to its neighboring points. If the distance exceeds a certain threshold, it is judged as a noise point and removed.

[0015] Vision sensor data acquisition

[0016] Use visual sensors such as cameras to collect image information of the surrounding environment; install multiple cameras with different viewing angles to obtain a more comprehensive image of the environment;

[0017] The image is preprocessed, including but not limited to grayscale, noise reduction and other operations; grayscale can reduce the amount of data and improve the subsequent processing speed; noise reduction uses the median filter method to remove noise in the image;

[0018] S2. Data time synchronization and spatial registration

[0019] Time Synchronization

[0020] To ensure the temporal consistency of data from different sensors, hardware synchronization or software synchronization is used: hardware synchronization uses a synchronization trigger signal to enable each sensor to start collecting data at the same time; software synchronization aligns the data based on the timestamps in the data from each sensor and adjusts the data to the same timeline;

[0021] If software synchronization is used, a master clock needs to be set, and the data of other sensors are adjusted according to the time difference with the master clock to ensure that the data of different sensors at the same time can correspond;

[0022] Spatial Registration

[0023] Determine the installation position and posture of each sensor on the navigation carrier, and establish the coordinate transformation relationship between the sensors by pre-calibration;

[0024] Convert the LiDAR point cloud data and visual image data to the world coordinate system of RTK positioning; given the translation vector and rotation matrix of the LiDAR relative to the RTK positioning device, convert the LiDAR point cloud data to the world coordinate system using the coordinate transformation formula;

[0025] S3. Feature extraction and fusion

[0026] RTK feature extraction

[0027] Extracting position, speed and direction feature information from RTK positioning data; the feature information reflects the absolute position and motion state of the navigation carrier in space;

[0028] LiDAR feature extraction

[0029] Segment the LiDAR point cloud data into different objects or regions. A distance-based segmentation method can be used to classify points with similar distances as the same object.

[0030] Extract geometric features of objects, including but not limited to edges, planes, and volumes of objects;

[0031] Visual feature extraction

[0032] Use image recognition algorithms to extract features from visual images, such as color, texture, and shape; use convolutional neural network (CNN) deep learning models to extract features from images;

[0033] Recognize specific visual signs, including but not limited to traffic signs and road markings; such signs can provide important reference information for navigation;

[0034] Feature Fusion

[0035] Use fusion algorithms to fuse the features extracted by RTK, LiDAR and visual sensors; the decision layer fusion, feature layer fusion or data layer fusion methods can be used;

[0036] In the feature layer fusion, a weight distribution fusion method based on neural network is adopted to assign different weights to the features of different sensors according to their reliability and accuracy;

[0037] S4. Environment modeling and path planning

[0038] Environmental Modeling

[0039] Construct an environmental model based on the fused feature data; it can be in the form of a grid map, a topological map or a three-dimensional model;

[0040] In a grid map, the environment is divided into small squares, and the sensor data is used to determine whether each square is occupied by an obstacle, thereby establishing an occupancy map of the environment; a topological map focuses on the key nodes and connection relationships in the environment, ignoring the specific geometric shapes;

[0041] Continuously update the environmental model. As the navigation carrier moves and the sensor data is updated, the environmental model is corrected in real time to reflect the changes in the environment.

[0042] Path Planning

[0043] Based on the constructed environment model, a path planning algorithm is used to determine the optimal path for the navigation carrier;

[0044] S5, Navigation Control and Feedback

[0045] Navigation Controls

[0046] Generate navigation control instructions based on the planned path and the current state of the navigation carrier; the control instructions include but are not limited to forward speed, steering angle, and braking force;

[0047] The PID controller feedback control algorithm is used to adjust the control instructions in real time according to the deviation between the actual position of the navigation carrier and the planned path, so that the navigation carrier can accurately travel along the planned path;

[0048] Feedback and Optimization

[0049] Continuously receive sensor data and monitor the navigation process in real time; if large navigation deviations or abnormalities are found, promptly adjust control instructions or re-plan the path;

[0050] According to the actual performance during navigation, the navigation algorithm and parameters are optimized; for example, the weight of feature fusion is adjusted and the parameters of the path planning algorithm are improved to improve the accuracy and efficiency of navigation.

[0051] Furthermore, the weighted fusion in the feature extraction and fusion in step S3 uses a multi-layer feedforward neural network (including an input layer, a hidden layer, and an output layer) to construct a weight evaluation model, and its basic principle is as follows:

[0052] 1. Input layer

[0053] The input vector x contains environmental parameters and some key measurement data of each sensor, which can be expressed as:

[0054] x=[x1,x2,…,x n ]

[0055] Among them, x1,x2,…,x m is an environmental parameter (e.g., x1 represents weather conditions, which can be numerically encoded, such as 1 for sunny, 2 for cloudy, 3 for rain, etc.; x2 represents light intensity, etc.), m+1 ,x m+2 ,…,x n is the relevant measurement data of each sensor (for example, for RTK sensors, x m+1is the positioning accuracy indicator; for lidar, x m+2 is the number of valid points in the point cloud data, etc. For visual sensors, x m+3 is an indicator of image clarity, etc.);

[0056] 2. Hidden layer calculation (taking one hidden layer as an example)

[0057] The input to the hidden layer neurons is:

[0058]

[0059] in, is the connection weight from the input layer to the hidden layer, is the bias of the jth neuron in the hidden layer, p is the number of neurons in the hidden layer; the output of the neurons in the hidden layer is transformed nonlinearly through the activation function:

[0060] a j =ReLU(z j )=max(0,z j )(j=1,2,…,p)

[0061] 3. Output layer calculation

[0062] The output layer corresponds to the weights of the three sensors, and the output vector is:

[0063] y=[w1,w2,w3]

[0064] The calculation method is:

[0065]

[0066] in, is the connection weight from the hidden layer to the output layer, is the bias of the kth neuron in the output layer;

[0067] At the same time, in order to ensure that the sum of the weights is 1, normalization can be performed, and the final weight calculation formula is:

[0068]

[0069] When training this neural network weight evaluation model, a large amount of labeled data (including input vectors x in different environments and corresponding samples of the optimal weight distribution of each sensor) is used to adjust the connection weights in the network by minimizing the loss function (such as the mean square error loss function). and bias The mean square error loss function can be expressed as:

[0070]

[0071] Where N is the number of training samples, y ik is the weight of the actual output of the kth sensor in the ith training sample, is the weight of the kth sensor predicted by the model; the network parameters are continuously updated through the optimization algorithm to minimize the loss function, thereby learning the ability to accurately assign weights according to different inputs.

[0072] Furthermore, the data collection in step S1 also includes, in the RTK data collection stage: in addition to preliminary screening of the positioning data to remove obvious outliers, it also further includes:

[0073] Use the historical data trend analysis method to make a secondary judgment on the suspected outliers; by analyzing the historical positioning data trend within the time period, if the current suspected outliers deviate greatly from the historical trend, conduct a more in-depth analysis and processing, and use the prediction model-based method to estimate the location coordinate value;

[0074] Regularly calibrate and test RTK positioning equipment to ensure its accuracy and stability; perform comparison tests with known high-precision position points, or use professional calibration tools to adjust equipment parameters.

[0075] In the LiDAR data collection phase: add a denoising method based on deep learning as a supplement; use a pre-trained deep neural network model to denoise the LiDAR point cloud data, learn the characteristics of different types of noise and effectively remove them;

[0076] After acquiring the 3D point cloud data, the data is compressed to reduce the pressure of data storage and transmission, and a lossless compression algorithm is used to balance the requirements of data quality and storage space;

[0077] In the visual sensor data acquisition step: when installing multiple cameras with different viewing angles, determine the best camera layout plan; consider the shape, size and operating environment of the navigation carrier, and determine the camera installation position and angle that can maximize coverage of the surrounding environment and avoid blind spots through simulation and experiments.

[0078] When preprocessing an image, in addition to grayscale and median filtering for noise reduction, it also includes image enhancement processing; the histogram equalization method is used to enhance the contrast of the image and improve the recognizability of the features in the image.

[0079] Furthermore, in the step S4 environmental modeling and path planning, based on the constructed environmental model, the Dijkstra algorithm is used to determine the optimal path of the navigation carrier; based on factors such as navigation targets, obstacle distribution, topography, etc., a safe and efficient path is planned; in path planning, obstacles should be avoided as much as possible, and routes with shorter distances and smoother traffic should be selected; when the environment changes, such as new obstacles appear or the target position changes, the path is replanned in time to ensure the accuracy and reliability of navigation.

[0080] The beneficial effects of the navigation method based on RTK laser vision fusion of the present invention are:

[0081] (1) By fusing the data of RTK, LiDAR and vision sensor, the present invention can give full play to the advantages of each sensor and make up for the limitations of a single sensor. RTK provides high-precision absolute position information, LiDAR accurately perceives the three-dimensional shape and distance of objects, and vision sensor recognizes specific signs and textures. The fusion of the three can improve the navigation system's estimation accuracy of position, speed and direction and reduce positioning errors.

[0082] In complex urban environments, RTK positioning may be blocked by signals from tall buildings, but lidar and visual sensors can assist in positioning by perceiving the surrounding environment, thereby improving the overall accuracy of navigation.

[0083] (2) The present invention has enhanced environmental perception capabilities. Through multi-sensor fusion, the navigation system can more comprehensively perceive the surrounding environment. The three-dimensional point cloud data of the laser radar can provide accurate distance and angle information of objects in the environment, helping to identify the shape and position of obstacles; the visual sensor can capture rich color, texture and sign information to enhance the understanding of the environment. This comprehensive environmental perception capability helps the navigation system better plan paths, avoid obstacles, and improve the safety and reliability of navigation.

[0084] In autonomous driving, the fused sensor data can enable the vehicle to more accurately identify traffic signs, pedestrians, other vehicles, etc., thereby making more reasonable decisions.

[0085] (3) The present invention can adapt to complex environmental changes, update the environmental model in real time, and adjust the path planning and control instructions in time according to the movement of the navigation carrier and the update of sensor data. When the environment changes, such as new obstacles appear or the target position changes, the system can quickly re-plan the path to ensure the accuracy and reliability of navigation.

[0086] For example, in logistics distribution, if there is road construction or traffic congestion, the navigation system can adjust the route in time to ensure that the goods can be delivered to the destination on time.

[0087] (4) The present invention can improve the reliability and stability of the system by screening, denoising, calibrating and other processes on sensor data, and using neural networks for weight distribution fusion and PID controllers for feedback control. It can reduce data errors and interference and improve the performance of the navigation system by removing outliers in RTK data, denoising lidar data, optimizing camera layout and image preprocessing and other operations.

[0088] At the same time, by continuously receiving sensor data for real-time monitoring and optimization, navigation deviations and abnormal situations can be discovered and handled in a timely manner to ensure the stable operation of the system.

[0089] (5) The navigation method based on RTK laser vision fusion of the present invention can achieve high-precision and high-reliability navigation in complex environments through multi-sensor fusion, optimized data processing and advanced algorithms, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] The present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0091] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0092] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0093] In order to make the technical solution of the present invention clearer, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0094] Reference Figure 1 , a navigation method based on RTK laser vision fusion, comprising the following steps: S1, data acquisition

[0095] RTK data collection

[0096] Install RTK positioning equipment, including a base station and a mobile station; the base station is set at a fixed point with a known precise position and continuously transmits signals containing position information; the mobile station is installed on the navigation carrier, receives the base station signal and satellite signals, and determines the carrier's high-precision position coordinates (longitude, latitude, altitude) and time information;

[0097] Perform a preliminary screening of the RTK data to remove obvious outliers (the positioning data at a certain moment is too different from the data at the previous and next moments, which can be temporarily identified as an outlier and marked for further processing);

[0098] LiDAR data collection

[0099] Start the laser radar to scan the surrounding environment and obtain three-dimensional point cloud data; the point cloud data contains the distance and angle information of objects in the environment;

[0100] De-noising the point cloud data: Statistical filtering is used to remove noise points caused by environmental interference or laser radar errors. Statistical filtering calculates the average distance from each point to its neighboring points. If the distance exceeds a certain threshold, it is judged as a noise point and removed.

[0101] Vision sensor data acquisition

[0102] Use visual sensors such as cameras to collect image information of the surrounding environment; install multiple cameras with different viewing angles to obtain a more comprehensive image of the environment;

[0103] The image is preprocessed, including but not limited to grayscale, noise reduction and other operations; grayscale can reduce the amount of data and improve the subsequent processing speed; noise reduction uses the median filter method to remove noise in the image;

[0104] S2. Data time synchronization and spatial registration

[0105] Time Synchronization

[0106] To ensure the temporal consistency of data from different sensors, hardware synchronization or software synchronization is used: hardware synchronization uses a synchronization trigger signal to enable each sensor to start collecting data at the same time; software synchronization aligns the data based on the timestamps in the data from each sensor and adjusts the data to the same timeline;

[0107] If software synchronization is used, a master clock needs to be set, and the data of other sensors are adjusted according to the time difference with the master clock to ensure that the data of different sensors at the same time can correspond;

[0108] Spatial Registration

[0109] Determine the installation position and posture of each sensor on the navigation carrier, and establish the coordinate transformation relationship between the sensors by pre-calibration;

[0110] Convert the LiDAR point cloud data and visual image data to the world coordinate system of RTK positioning; given the translation vector and rotation matrix of the LiDAR relative to the RTK positioning device, convert the LiDAR point cloud data to the world coordinate system using the coordinate transformation formula;

[0111] S3. Feature extraction and fusion

[0112] RTK feature extraction

[0113] Extracting position, speed and direction feature information from RTK positioning data; the feature information reflects the absolute position and motion state of the navigation carrier in space;

[0114] LiDAR feature extraction

[0115] Segment the LiDAR point cloud data into different objects or regions. A distance-based segmentation method can be used to classify points with similar distances as the same object.

[0116] Extract geometric features of objects, including but not limited to edges, planes, and volumes of objects;

[0117] Visual feature extraction

[0118] Use image recognition algorithms to extract features from visual images, such as color, texture, and shape; use convolutional neural network (CNN) deep learning models to extract features from images;

[0119] Recognize specific visual signs, including but not limited to traffic signs and road markings; such signs can provide important reference information for navigation;

[0120] Feature Fusion

[0121] Use fusion algorithms to fuse the features extracted by RTK, LiDAR and visual sensors; the decision layer fusion, feature layer fusion or data layer fusion methods can be used;

[0122] In the feature layer fusion, a weight distribution fusion method based on neural network is adopted to assign different weights to the features of different sensors according to their reliability and accuracy;

[0123] S4. Environment modeling and path planning

[0124] Environmental Modeling

[0125] Construct an environmental model based on the fused feature data; it can be in the form of a grid map, a topological map or a three-dimensional model;

[0126] In a grid map, the environment is divided into small squares, and the sensor data is used to determine whether each square is occupied by an obstacle, thereby establishing an occupancy map of the environment; a topological map focuses on the key nodes and connection relationships in the environment, ignoring the specific geometric shapes;

[0127] Continuously update the environmental model. As the navigation carrier moves and the sensor data is updated, the environmental model is corrected in real time to reflect the changes in the environment.

[0128] Path Planning

[0129] Based on the constructed environment model, a path planning algorithm is used to determine the optimal path for the navigation carrier;

[0130] S5, Navigation Control and Feedback

[0131] Navigation Controls

[0132] Generate navigation control instructions based on the planned path and the current state of the navigation carrier; the control instructions include but are not limited to forward speed, steering angle, and braking force;

[0133] The PID controller feedback control algorithm is used to adjust the control instructions in real time according to the deviation between the actual position of the navigation carrier and the planned path, so that the navigation carrier can accurately travel along the planned path;

[0134] Feedback and Optimization

[0135] Continuously receive sensor data and monitor the navigation process in real time; if large navigation deviations or abnormalities are found, promptly adjust control instructions or re-plan the path;

[0136] According to the actual performance during navigation, the navigation algorithm and parameters are optimized; for example, the weight of feature fusion is adjusted and the parameters of the path planning algorithm are improved to improve the accuracy and efficiency of navigation.

[0137] Preferably, the weighted fusion in the feature extraction and fusion in step S3 uses a multi-layer feedforward neural network (including an input layer, a hidden layer and an output layer) to construct a weight evaluation model, and its basic principle is as follows:

[0138] 1. Input layer

[0139] The input vector x contains environmental parameters and some key measurement data of each sensor, which can be expressed as:

[0140] x=[x1,x2,…,x n ]

[0141] Among them, x1,x2,…,x m are environmental parameters (x1 represents weather conditions, which can be numerically coded, such as sunny for 1, cloudy for 2, rainy for 3, etc.; x2 represents light intensity, etc.), m+1 ,x m+2 ,…,x n is the relevant measurement data of each sensor (for RTK sensor, x m+1 is the positioning accuracy indicator; for lidar, x m+2 is the number of valid points in the point cloud data; for visual sensors, x m+3 is an indicator of image clarity);

[0142] 2. Hidden layer calculation (taking one hidden layer as an example)

[0143] The input to the hidden layer neurons is:

[0144]

[0145] in, is the connection weight from the input layer to the hidden layer, is the bias of the jth neuron in the hidden layer, p is the number of neurons in the hidden layer; the output of the neurons in the hidden layer is transformed nonlinearly through the activation function:

[0146] a j =ReLU(z j )=max(0,z j )(j=1,2,…,p)

[0147] 3. Output layer calculation

[0148] The output layer corresponds to the weights of the three sensors, and the output vector is:

[0149] y=[w1,w2,w3]

[0150] The calculation method is:

[0151]

[0152] in, is the connection weight from the hidden layer to the output layer, is the bias of the kth neuron in the output layer;

[0153] At the same time, in order to ensure that the sum of the weights is 1, normalization can be performed, and the final weight calculation formula is:

[0154]

[0155] When training this neural network weight evaluation model, a large amount of labeled data (including input vectors x in different environments and corresponding samples of the optimal weight distribution of each sensor) is used to adjust the connection weights in the network by minimizing the loss function (such as the mean square error loss function). and bias The mean square error loss function can be expressed as:

[0156]

[0157] Where N is the number of training samples, y ik is the weight of the actual output of the kth sensor in the ith training sample, is the weight of the kth sensor predicted by the model; the network parameters are continuously updated through the optimization algorithm to minimize the loss function, thereby learning the ability to accurately assign weights according to different inputs.

[0158] Preferably, the data collection in step S1 further includes, in the RTK data collection stage: in addition to preliminary screening of the positioning data to remove obvious outliers, further including:

[0159] Use the historical data trend analysis method to make a secondary judgment on the suspected outliers; by analyzing the historical positioning data trend within the time period, if the current suspected outliers deviate greatly from the historical trend, conduct a more in-depth analysis and processing, and use the prediction model-based method to estimate the location coordinate value;

[0160] Regularly calibrate and test RTK positioning equipment to ensure its accuracy and stability; perform comparison tests with known high-precision position points, or use professional calibration tools to adjust equipment parameters.

[0161] In the LiDAR data collection phase: add a denoising method based on deep learning as a supplement; use a pre-trained deep neural network model to denoise the LiDAR point cloud data, learn the characteristics of different types of noise and effectively remove them;

[0162] After acquiring the 3D point cloud data, the data is compressed to reduce the pressure of data storage and transmission, and a lossless compression algorithm is used to balance the requirements of data quality and storage space;

[0163] In the visual sensor data acquisition step: when installing multiple cameras with different viewing angles, determine the best camera layout plan; consider the shape, size and operating environment of the navigation carrier, and determine the camera installation position and angle that can maximize coverage of the surrounding environment and avoid blind spots through simulation and experiments.

[0164] When preprocessing an image, in addition to grayscale and median filtering for noise reduction, it also includes image enhancement processing; the histogram equalization method is used to enhance the contrast of the image and improve the recognizability of the features in the image.

[0165] Preferably, in the step S4 environmental modeling and path planning, based on the constructed environmental model, the Dijkstra algorithm is used to determine the optimal path of the navigation carrier; based on factors such as navigation targets, obstacle distribution, topography, etc., a safe and efficient path is planned; in the path planning, obstacles should be avoided as much as possible, and routes with shorter distances and smoother traffic should be selected; when the environment changes, such as new obstacles appear or the target position changes, the path is replanned in time to ensure the accuracy and reliability of navigation.

[0166] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A navigation method based on RTK laser vision fusion, characterized in that: The following steps are included: S1. Data collection RTK data collection Install RTK positioning equipment, including a base station and a mobile station; the base station is set at a fixed point with a known precise position and continuously transmits signals containing position information; the mobile station is installed on the navigation carrier, receives the base station signal and satellite signals, and determines the high-precision position coordinates and time information of the carrier; Perform preliminary screening of RTK data to remove obvious outliers; LiDAR data collection Start the laser radar to scan the surrounding environment and obtain three-dimensional point cloud data; the point cloud data contains the distance and angle information of objects in the environment; De-noising the point cloud data: using statistical filtering to remove noise points caused by environmental interference or errors in the lidar itself; Vision sensor data acquisition Use visual sensors such as cameras to collect image information of the surrounding environment; install multiple cameras with different viewing angles to obtain a more comprehensive image of the environment; The image is preprocessed, including but not limited to grayscale, noise reduction and other operations; grayscale can reduce the amount of data and improve the subsequent processing speed; noise reduction uses the median filter method to remove noise in the image; S2. Data time synchronization and spatial registration Time Synchronization To ensure the temporal consistency of data from different sensors, hardware synchronization or software synchronization is used: Hardware synchronization uses a synchronization trigger signal to enable each sensor to start collecting data at the same time; Software synchronization aligns the data to the same timeline based on the timestamps in each sensor data. Spatial Registration Determine the installation position and posture of each sensor on the navigation carrier, and establish the coordinate transformation relationship between the sensors by pre-calibration; Convert the LiDAR point cloud data and visual image data to the world coordinate system of RTK positioning; given the translation vector and rotation matrix of the LiDAR relative to the RTK positioning device, convert the LiDAR point cloud data to the world coordinate system using the coordinate transformation formula; S3. Feature extraction and fusion RTK feature extraction Extracting position, speed and direction feature information from RTK positioning data; the feature information reflects the absolute position and motion state of the navigation carrier in space; LiDAR feature extraction Segment the LiDAR point cloud data into different objects or regions. A distance-based segmentation method can be used to classify points with similar distances as the same object. Extract geometric features of objects, including but not limited to edges, planes, and volumes of objects; Visual feature extraction Use image recognition algorithms to extract features from visual images, such as color, texture, and shape; use convolutional neural network deep learning models to extract features from images; Recognize specific visual signs, including but not limited to traffic signs and road markings; such signs can provide important reference information for navigation; Feature Fusion Use fusion algorithms to fuse the features extracted by RTK, LiDAR and visual sensors; the decision layer fusion, feature layer fusion or data layer fusion methods can be used; In the feature layer fusion, a weight distribution fusion method based on neural network is adopted to assign different weights to the features of different sensors according to their reliability and accuracy; S4. Environment modeling and path planning Environmental Modeling Construct an environmental model based on the fused feature data; it can be in the form of a grid map, a topological map or a three-dimensional model; In a grid map, the environment is divided into small squares, and the sensor data is used to determine whether each square is occupied by an obstacle, thereby establishing an occupancy map of the environment; a topological map focuses on the key nodes and connection relationships in the environment, ignoring the specific geometric shapes; Continuously update the environmental model. As the navigation carrier moves and the sensor data is updated, the environmental model is corrected in real time to reflect changes in the environment; Path Planning Based on the constructed environment model, a path planning algorithm is used to determine the optimal path for the navigation carrier; S5, Navigation Control and Feedback Navigation Controls Generate navigation control instructions based on the planned path and the current state of the navigation carrier; the control instructions include but are not limited to forward speed, steering angle, and braking force; The PID controller feedback control algorithm is used to adjust the control instructions in real time according to the deviation between the actual position of the navigation carrier and the planned path, so that the navigation carrier can accurately travel along the planned path; Feedback and Optimization Continuously receive sensor data and monitor the navigation process in real time; If the navigation deviation is large or an abnormal situation occurs, adjust the control instructions or re-plan the path in time; According to the actual performance during navigation, the navigation algorithm and parameters are optimized; for example, the weight of feature fusion is adjusted and the parameters of the path planning algorithm are improved to improve the accuracy and efficiency of navigation.

2. According to claim 1, a navigation method based on RTK laser vision fusion is characterized in that: The weighted fusion in step S3 feature extraction and fusion uses a multi-layer feedforward neural network to construct a weight evaluation model, and its basic principle is as follows:

1. Input layer The input vector x contains environmental parameters and some key measurement data of each sensor, which can be expressed as: x=[x1,x2,…,x n ] Among them, x1, x2, …, x m are environmental parameters (x1 represents weather conditions, which are numerically encoded, such as 1 for sunny, 2 for cloudy, 3 for rain, etc.; x2 represents light intensity, etc.), m+1 , x m+2 , …, x n is the relevant measurement data of each sensor (for RTK sensor, x m+1 is the positioning accuracy indicator; for lidar, x m+2 is the number of valid points in the point cloud data; for visual sensors, x m+3 is an indicator of image clarity); 2. Hidden layer calculation The input to the hidden layer neurons is: in, is the connection weight from the input layer to the hidden layer, is the bias of the jth neuron in the hidden layer, p is the number of neurons in the hidden layer; the output of the neurons in the hidden layer is transformed nonlinearly through the activation function: aj=ReLU(z j )=max(0,z j (j=1, 2,…, p) 3. Output layer calculation The output layer corresponds to the weights of the three sensors, and the output vector is: y=[w1,w2,w3] The calculation method is: in, is the connection weight from the hidden layer to the output layer, is the bias of the kth neuron in the output layer; At the same time, in order to ensure that the sum of the weights is 1, normalization can be performed, and the final weight calculation formula is: When training this neural network weight evaluation model, a large amount of labeled data is used to adjust the connection weights in the network by minimizing the loss function. and bias The mean square error loss function can be expressed as: Where N is the number of training samples, y ik is the weight of the actual output of the kth sensor in the ith training sample, is the weight of the kth sensor predicted by the model; the network parameters are continuously updated through the optimization algorithm to minimize the loss function, thereby learning the ability to accurately assign weights according to different inputs.

3. The navigation method based on RTK laser vision fusion according to claim 1 is characterized in that: The step S1 of data collection also includes, in the RTK data collection stage: in addition to preliminary screening of the positioning data to remove obvious outliers, it also further includes: Use the historical data trend analysis method to make a secondary judgment on the suspected outliers; by analyzing the historical positioning data trend within the time period, if the current suspected outliers deviate greatly from the historical trend, conduct a more in-depth analysis and processing, and use the prediction model-based method to estimate the location coordinate value; Regularly calibrate and test RTK positioning equipment to ensure its accuracy and stability; perform comparison tests with known high-precision position points, or use professional calibration tools to adjust equipment parameters. In the LiDAR data collection phase: add a denoising method based on deep learning as a supplement; use a pre-trained deep neural network model to denoise the LiDAR point cloud data, learn the characteristics of different types of noise and effectively remove them; After acquiring the 3D point cloud data, the data is compressed to reduce the pressure of data storage and transmission, and a lossless compression algorithm is used to balance the requirements of data quality and storage space; In the visual sensor data acquisition step: when installing multiple cameras with different viewing angles, determine the best camera layout plan; consider the shape, size and operating environment of the navigation carrier, and determine the camera installation position and angle that can maximize coverage of the surrounding environment and avoid blind spots through simulation and experiments. When preprocessing an image, in addition to grayscale and median filtering for noise reduction, it also includes image enhancement processing; the histogram equalization method is used to enhance the contrast of the image and improve the recognizability of the features in the image.

4. The navigation method based on RTK laser vision fusion according to claim 1 is characterized in that: In the step S4, the environmental modeling and path planning uses the Dijkstra algorithm to determine the optimal path of the navigation carrier based on the constructed environmental model; a safe and efficient path is planned based on factors such as navigation target, obstacle distribution, topography, etc.; When planning a route, you should try to avoid obstacles and choose routes that are shorter and have smoother traffic. When the environment changes, such as when new obstacles appear or the target location changes, you should re-plan the route in a timely manner to ensure the accuracy and reliability of navigation.

Images