An autonomous road and bridge marking system based on multi-sensor data fusion and stereo vision control

Through the integration of a variety of sensor data fusion systems with lidar and binocular vision sensing devices, the problem of insufficient positioning accuracy of the automatic scribing machine is solved, error fusion and compensation are achieved, and the accuracy and stability of scribing are improved.

CN114942641BActive Publication Date: 2024-11-22ZHONGKAI UNIV OF AGRI & ENG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210634018.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-06
Publication Date
2024-11-22
Estimated Expiration
2042-06-06

AI Technical Summary

Technical Problem

The existing automatic scribing machine is limited in positioning accuracy on highways, and the error is large, resulting in inaccurate scribing. This is mainly due to the superposition of errors in Beidou navigation system and errors in the scribing mechanism, and the lack of three-dimensional fusion of multiple sensor data. Vision control system.

Method used

Using a variety of sensor data fusion stereo vision control systems, the controller integrates lidar and binocular vision sensing devices to realize automatic navigation and obstacle avoidance of the scribe car, and adjust the motion trajectory through error fusion and compensation to ensure the scribe accuracy.

Benefits of technology

It improves the positioning accuracy and application ease of the marking machine, reduces errors, and ensures the accuracy and stability of the marking, especially in highway environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114942641B_ABST
    Figure CN114942641B_ABST
Patent Text Reader

Abstract

A road and bridge autonomous walking marking system with multiple sensor data fusion stereo vision control, which includes: a marking vehicle, multiple groups of laser radar sensor devices, multiple groups of stereo vision control and controller, the controller includes a marking vehicle automatic navigation module and a marking control module; the marking vehicle automatic navigation module and the marking control module transmit signals with the help of a ROS intermediate communication module; when the second laser radar sensor device and the second binocular vision sensor device detect that there is a cumulative error between the line drawn by the marking machine and the line given by the navigation, feedback is given to the controller, and the controller instructs the terminal mechanism to make adjustments to correct the deviation. The control connection design is reasonable, and the controller controls multiple groups of laser radar sensor devices and binocular vision sensor devices to be integrated into one, which can detect the marking of the marking vehicle, the automatic marking vehicle forward navigation and obstacle avoidance, and the positioning of the marking vehicle relative to the cement railing on the side of the highway; the error fusion and compensation are used to adjust the movement trajectory of the marking machine.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a road and bridge autonomous walking and marking system controlled by stereoscopic vision and fusion of multiple sensor data. Background Art

[0002] According to market research and relevant public literature searches, the marking of urban and rural roads, highways and bridges is currently done manually. Although existing patent documents and some technical achievement documents record that the independent application technologies of automatic marking, ranging laser radar and stereo vision sensing technology are relatively mature, they have not been found to be applied in automatic highway marking machines.

[0003] The walking route of the automatic road marking machine is consistent with the marking route. If the Beidou (GPS) navigation system is used, the positioning accuracy is still at the centimeter level. Especially on highways, the positioning accuracy is affected by high-rise buildings, mountains and high objects on the roadside, and the interference of clouds and weather in the sky. The error is at least 20 cm-50 cm. This problem has become a problem that plagues the actual engineering application of automatic road marking machines. In fact, there are two kinds of errors when marking. One is the error of Beidou navigation when the machine is walking, and the other is the error generated when the end mechanism of the road marking machine draws the line. The addition of the two forms a comprehensive error that makes the marking inaccurate. For example, the literature describes the use of machine vision to find the waterline to achieve precise positioning ([1] Shi Lei, Highway Unmanned Road Marking Vehicle. Shandong Province, Shandong Jiaotong University, 2020-12-30. It mainly uses image recognition to guide the road marking equipment to achieve posture adjustment and automatic driving along the preset route; similarly, there are inevitably navigation detection errors, and the system control settings are not reasonable. It has not yet been able to disclose a multi-sensor data fusion stereoscopic vision control road and bridge autonomous walking road marking system.

[0004] To this end, we have developed an autonomous road and bridge marking system that uses multiple sensor data fusion and stereo vision control. Summary of the invention

[0005] The technical problem to be solved by the purpose of the present invention is to provide an autonomous road and bridge marking system with stereo vision control and multiple sensor data fusion. The control connection design is reasonable, and multiple groups of laser radar sensor devices and binocular vision sensor devices are integrated into one through a controller to implement the detection of marked lines, automatic forward navigation and obstacle avoidance of the marking vehicle, and positioning of the marking vehicle relative to the concrete railings on the side of the highway; error fusion and compensation are performed to adjust the motion trajectory of the marking machine; thereby making the marking vehicle easy to implement and apply.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is to provide a road and bridge autonomous walking and marking system with multiple sensor data fusion and stereoscopic vision control, which is characterized in that it includes:

[0007] Line striping car;

[0008] a first laser radar sensor device and a first binocular vision sensor device, and

[0009] A second laser radar sensor device and a second binocular vision sensor device;

[0010] A controller, which is in communication connection with the first laser radar sensor device, the first binocular vision sensor device, the second laser radar sensor device and the second binocular vision sensor device; the controller includes an automatic navigation module for a marking vehicle and a marking control module; the automatic navigation module for a marking vehicle and the marking control module use the ROS intermediate communication module to interactively link the walking control, the terminal mechanism marking control, the first laser radar sensor device, the first binocular vision sensor device, the second laser radar sensor device and the second binocular vision sensor device to issue instructions;

[0011] When the second laser radar sensor device and the second binocular vision sensor device detect that there is a cumulative error between the line drawn by the marking machine and the line given by the navigation, feedback is given to the controller, and the controller instructs the end mechanism to make adjustments to correct the deviation.

[0012] In one or more embodiments of the present invention, the automatic navigation module of the marking vehicle includes a positioning module and a path module, and the automatic navigation module of the marking vehicle includes a positioning module and a path module that are bidirectionally connected to the ROS intermediate communication module.

[0013] In one or more embodiments of the present invention, the marking control module includes a point cloud algorithm reconstruction module and an end mechanism; the point cloud algorithm reconstruction module and the end mechanism are bidirectionally connected to the ROS intermediate communication module.

[0014] In one or more embodiments of the present invention, the first laser radar sensor device and the first binocular vision sensor device are used for forward navigation and obstacle avoidance of the automatic lane marking vehicle.

[0015] In one or more embodiments of the present invention, the second laser radar sensor device and the second binocular vision sensor device are used to locate the relative position of the marking vehicle and the concrete guardrail on the side of the road.

[0016] In one or more embodiments of the present invention, the second binocular vision sensor device is used to detect the size of the drawn line by the end mechanism of the marking machine and feed it back to the controller;

[0017] When the cumulative error between the detected line and the line given by the navigation is within a set range, the terminal mechanism is adjusted to correct the deviation in real time;

[0018] When the cumulative error between the detected line and the line given by the navigation exceeds the set range, the error data is transmitted to the laser sensor, and the path of the vehicle is controlled by the controller through error fusion and compensation.

[0019] In one or more embodiments of the present invention, the second binocular vision sensing device detects the width of the marking and the distance between the center line of the width and the left railing ahead on the marking, and the second lidar sensing device measures the corresponding width and the distance between the center line of the width and the left railing ahead and feeds them back to the controller for comparison to determine whether it is within the tolerance range; if there is an error, the error data is sent to the controller for correction.

[0020] Compared with the background technology, the present invention has the following effects:

[0021] Since the present invention adopts the above scheme, the control connection design is reasonable, and multiple groups of laser radar sensor devices and binocular vision sensor devices are integrated into one through the controller to detect the marking of the marking vehicle, automatically navigate and avoid obstacles, and locate the relative position of the marking vehicle and the concrete railings on the side of the highway; the error fusion and compensation are carried out to adjust the motion trajectory of the marking machine; thereby making the marking vehicle easy to implement and apply. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of a road and bridge autonomous walking and marking system controlled by stereo vision and fusion of multiple sensor data in one embodiment of the present invention;

[0023] Figure 2 A control integration block diagram of a road and bridge autonomous walking and marking system controlled by stereo vision and fusion of multiple sensor data in one embodiment of the present invention;

[0024] Figure 3 It is a schematic diagram of the marking state of a road and bridge autonomous walking marking system controlled by stereo vision and fusion of multiple sensor data in one embodiment of the present invention during marking application;

[0025] Those skilled in the art can understand the shapes, structures and schemes obtained from the figures shown in the drawings. The various components of the drawings are not necessarily to scale, and the sizes of the various components and elements of the drawings may be enlarged or reduced to more clearly illustrate the embodiments of the present invention described herein. DETAILED DESCRIPTION

[0026] The following references Figure 1-2To describe a specific embodiment, however, those skilled in the art will easily understand that the detailed description given herein with respect to these drawings is for illustrative purposes only and should not be understood as being restrictive. This embodiment provides a road and bridge autonomous walking and marking system with multiple sensor data fusion and stereoscopic vision control, which includes: a marking vehicle 1, a first laser radar sensor device 2, a first binocular vision sensor device 3, a second laser radar sensor device 4, a second binocular vision sensor device 5 and a controller 6, the controller 6 is communicatively connected with the first laser radar sensor device 2, the first binocular vision sensor device 3, the second laser radar sensor device 4 and the second binocular vision sensor device 5; the controller 5 includes a marking vehicle automatic navigation module and a marking control module; the marking vehicle automatic navigation module and the marking control module interact with each other through the ROS intermediate communication module to issue instructions to the walking control, terminal mechanism marking control, the first laser radar sensor device, the first binocular vision sensor device, the second laser radar sensor device and the second binocular vision sensor device included in the marking vehicle.

[0027] When the second laser radar sensor device and the second binocular vision sensor device detect that there is a cumulative error between the line drawn by the marking machine and the line given by the navigation, feedback is given to the controller 6, and the controller instructs the end mechanism 7 to make adjustments to correct the deviation.

[0028] The automatic navigation module of the marking vehicle includes a positioning module and a path module, and the automatic navigation module of the marking vehicle includes a positioning module and a path module that are bidirectionally connected to the ROS intermediate communication module.

[0029] The line marking control module includes a point cloud algorithm reconstruction module and a terminal mechanism; the point cloud algorithm reconstruction module and the terminal mechanism are bidirectionally connected to the ROS intermediate communication module. The first laser radar sensor device and the first binocular vision sensor device are used for the automatic line marking vehicle to navigate forward and avoid obstacles. The second laser radar sensor device and the second binocular vision sensor device are used to locate the relative position between the line marking vehicle and the concrete railing on the side of the road.

[0030] The second binocular vision sensor is used to detect the size of the drawn line of the marking machine's terminal mechanism and feed it back to the controller; when the cumulative error between the detected marked line and the line given by the navigation is within the set range, the terminal mechanism is adjusted to correct the deviation in real time; when the cumulative error between the detected marked line and the line given by the navigation exceeds the set range, the error data is transmitted to the laser sensor, and the path of the controller is controlled through error fusion and compensation. The second binocular vision sensor detects the width of the marked line, the distance between the center line of the width and the front left railing during marking, and feeds back the corresponding width and the distance between the center line of the width and the front left railing measured by the second laser radar sensor to the controller for comparison to determine whether it is within the tolerance range; if there is an error, the error data is sent to the controller for correction. In this way, two groups of laser radars are used to provide data for the autonomous forward navigation and body positioning of the road marking machine 1. The first group is used for the forward navigation of the automatic road marking vehicle, and the second group is used for the precise positioning of the relative position between the road marking vehicle and the concrete railings on the side of the highway. The laser radar model with a detection accuracy of 2mm can be selected for implementation. The relative positioning ensures that the marking line is parallel to the railing, and can also meet the requirements of straight lines and arcs; at the same time, two groups of stereo vision are used. The first group of binoculars is used for walking route navigation and real-time obstacle avoidance. Its data is fused with the first group of laser radar data to achieve precise navigation and obstacle avoidance. The binocular vision of the second binocular vision sensor device can detect the size of the drawn line of the end mechanism of the road marking machine. When there is a cumulative error between the detected drawn line and the line given by the navigation, the visual control of the end mechanism of the road marking machine is slightly adjusted to correct the deviation in real time. When the error exceeds the slight adjustment amount of the mechanism, the large error data is transmitted to the laser sensor. Through error fusion and compensation, the path of the controller walking vehicle is controlled.

[0031] In the specific implementation, the binocular vision sensor device can preferably use the YOLOV5 algorithm to analyze the road, the railings on both sides, obstacles, organisms, and images to accurately identify the road ahead, ensure that the relative distance L1 between the car and the railing remains unchanged, and identify overtaking, sidewalks, and road obstacles. The dimensions of the lines drawn by the second binocular vision sensor device, including the width of the lines drawn, the distance between the center line of the calculated width and the front left railing within the marking tolerance range, can be compared with the dimensions measured by the second laser radar sensor device 4. If there is an error, the error data is sent to the control system for correction; the height of the terminal marking mechanism and the ground is required to maintain a height design value. If it is within the tolerance range, the control remains unchanged. If it is too high or too low, the controller makes adjustments.

[0032] like Figure 3As shown, when the end marking mechanism finishes drawing a line and returns, when drawing the second line, the first binocular vision sensor device 3 is used to detect the drawn line, and the distance data of the railing measured by the second binocular vision sensor device 5 is calculated and fused; the YOLOV5 algorithm can be used for multi-target classification and recognition, and then the distance between the center line of the first drawn line and the center line of the current drawn line is detected and the relative position of the vehicle is controlled in real time, and the relative position calculation is fused with the laser radar data; the classification and recognition algorithm can classify objects such as lines, obstacles, vehicles and creatures; and then the obstacle avoidance path planning is carried out through the fusion of vision and laser radar data. In this way, the control connection design is reasonable, and the controller controls multiple groups of laser radar sensors and binocular vision sensors to implement the detection of the marking of the marking vehicle, the automatic marking vehicle forward navigation and obstacle avoidance, and the positioning of the relative position between the marking vehicle and the cement railing on the side of the highway; the error fusion and compensation are used to adjust the motion trajectory of the marking machine; so that the marking vehicle is easy to implement.

[0033] For the purpose of illustration, the foregoing description uses specific nomenclature to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that specific details are not required in order to practice the embodiments. Those skilled in the art should understand that the present invention is not limited to the specific embodiments described above, and improvements and substitutions based on the present invention using techniques known in the art fall within the scope of the present invention and should be defined by the claims.

Claims

1. An autonomous road and bridge marking system with multiple sensor data fusion and stereo vision control. It is characterized in that It includes: Line striping car; A first laser radar sensor device and a first binocular vision sensor device are provided on the marking vehicle, and A second laser radar sensor device and a second binocular vision sensor device; A controller, which is in communication connection with the first laser radar sensor device, the first binocular vision sensor device, the second laser radar sensor device and the second binocular vision sensor device; the controller includes an automatic navigation module for a marking vehicle and a marking control module; the automatic navigation module for a marking vehicle and the marking control module use the ROS intermediate communication module to interactively link the walking control, the terminal mechanism marking control, the first laser radar sensor device, the first binocular vision sensor device, the second laser radar sensor device and the second binocular vision sensor device to issue instructions; When the second laser radar sensor device and the second binocular vision sensor device detect that there is a cumulative error between the line drawn by the marking machine and the line given by the navigation, feedback is given to the controller, and the controller instructs the end mechanism to make adjustments to correct the deviation; The first laser radar sensor device and the first binocular vision sensor device are used for forward navigation and obstacle avoidance of the automatic lane marking vehicle; The second laser radar sensor device and the second binocular vision sensor device are used to locate the relative position between the marking vehicle and the concrete railings on the roadside; the second binocular vision sensor device is used to detect the size of the drawn line by the end mechanism of the marking machine and feed it back to the controller; When the cumulative error between the detected line and the line given by the navigation is within a set range, the terminal mechanism is adjusted to correct the deviation in real time; When the cumulative error between the detected line and the line given by the navigation exceeds the set range, the error data is transmitted to the second laser sensor, and the path of the vehicle is controlled by the controller through error fusion and compensation.

2. The road and bridge autonomous walking and marking system controlled by multiple sensor data fusion and stereo vision according to claim 1, Features: The automatic navigation module of the marking vehicle includes a positioning module and a path module, and the automatic navigation module of the marking vehicle includes a positioning module and a path module that are bidirectionally connected to the ROS intermediate communication module.

3. The road and bridge autonomous walking and marking system controlled by stereo vision and fusion of multiple sensor data according to claim 2, Features: The line marking control module includes a point cloud algorithm reconstruction module and a terminal mechanism; the point cloud algorithm reconstruction module and the terminal mechanism are bidirectionally connected to the ROS intermediate communication module.

4. The road and bridge autonomous walking and marking system controlled by stereo vision and fusion of multiple sensor data according to claim 3, Features: The second binocular vision sensing device detects the width of the marking and the distance between the center line of the width and the left front railing on the marking, and the second laser radar sensing device measures the corresponding width and the distance between the center line of the width and the left front railing, and feeds them back to the controller for comparison to determine whether it is within the tolerance range; if there is an error, the error data is sent to the controller for correction.

5. The road and bridge autonomous walking and marking system controlled by stereo vision and fusion of multiple sensor data according to claim 4, Features: The height of the terminal mechanism and the ground is maintained at a designed height value. If a height deviation occurs, the second binocular vision sensor device feeds back to the controller for adjustment.