Situational awareness method and system for assisted remote control driving of ground unmanned platforms

Abstract

The present invention relates to a situational awareness method and system for assisted remote control driving of ground unmanned platforms, and belongs to the technical field of remote control of ground unmanned systems. The present invention carries an environmental perception sensor on the unmanned platform to perform situational awareness such as traversable area detection and dangerous object identification on the remote driving environment of the unmanned platform, and superimposes situational awareness information on video images to assist remote control driving, improve the safety of remote control driving, reduce driving fatigue of operators, enhance the friendliness of human-computer interaction, and improve the driving experience of remote control of unmanned platforms.

Description

Technical Field

[0001] The present invention belongs to the technical field of assisted remote control driving in remote control of ground unmanned systems, and specifically relates to a situation awareness method and system for assisted remote control driving of ground unmanned platforms. Background Art

[0002] Remote control technology for ground unmanned systems is an important human-computer interaction technology. With the help of radio communication and video feedback, it provides the driver with environmental information of the remote-controlled unmanned platform. The driver makes driving decisions based on the feedback information and controls the remote unmanned platform. In the traditional remote control process, the driver is required to manually judge the passable area of ​​the road within a limited field of view, avoid static positive obstacles such as trees and roadblocks, and negative obstacles such as potholes, and effectively dodge dynamic obstacles such as pedestrians and vehicles. The existing remote control of ground unmanned systems has high requirements on the driver's driving quality. It is necessary to analyze the driving environment within a limited image field of view and low image resolution, and the driving experience is poor, which is specifically reflected in the following aspects:

[0003] 1) Affected by the remote control radio communication delay and video bandwidth, the remote video resolution is low. At the same time, due to factors such as communication obstruction, problems such as video delay and unclear picture will occur, and the human-computer interaction is poor, which affects the remote driving decision of the operator.

[0004] 2) Video images are greatly affected by changes in lighting, especially at tunnel exits and in front of the light, where the images are blurry and unusable, posing a safety hazard to remote control.

[0005] 3) The visible field of vision of the remote-controlled driver is greatly affected by the vehicle-mounted camera and the installation position. Obstacles near and on both sides of the vehicle can easily enter the blind spot, affecting driving safety.

[0006] 4) The driver needs to maintain a highly tense driving state at all times in order to analyze the surrounding driving environment, especially to prevent sudden appearance of dynamic obstacles such as pedestrians and vehicles, which can easily cause fatigue driving.

[0007] Designing assisted remote control driving technology, with the help of unmanned platform on-board sensors, can conduct autonomous situational awareness of the surrounding environment, provide hazardous object identification information and road passable areas, and superimpose them on video images. This is very necessary for improving the decision-making ability of remote operators and improving driving safety. Summary of the invention

[0008] 1. Technical issues to be resolved

[0009] The technical problem to be solved by the present invention is: how to design a situation awareness method and a vehicle-mounted system for assisted remote control driving of ground unmanned platforms, so as to solve the problems of poor human-computer interaction, high driving risk, and high dependence on driver quality in the existing remote control technology that only relies on the remote control driver to perform environmental situation analysis.

[0010] (II) Technical solution

[0011] In order to solve the above technical problems, the present invention provides a situation awareness method for assisted remote control driving of ground unmanned platforms. The method is implemented using a vehicle-mounted system, which includes a three-dimensional laser radar and a color camera. In the method, the three-dimensional laser radar is used to obtain three-dimensional point cloud information around the unmanned platform's driving environment, and positive and negative obstacles and passable areas are detected; and through the joint calibration of the three-dimensional laser radar and the color camera, the detection results are superimposed on the video image to provide the remote control driver with auxiliary information such as obstacle warning information and passable areas; and the color camera is used to obtain visual information of the unmanned platform's driving environment, and dynamic obstacles such as pedestrians and vehicles are identified and marked on the image through deep learning-based model training.

[0012] Preferably, the positive obstacle detection based on the three-dimensional point cloud information of the three-dimensional laser radar and the positive obstacle detection result are superimposed on the video image through the joint calibration of the three-dimensional laser radar and the color camera. The specific process includes:

[0013] Step 1: Obtain 3D radar point cloud data of the remote driving environment of the ground unmanned platform;

[0014] Step 2: Rasterize the 3D radar point cloud data, project the 3D radar point cloud data onto a 2D overhead grid map, and count the number of point clouds and height information in each grid;

[0015] Step 3: Set the n neighborhood of each grid as the reference basis for the ground height, and take the lowest height of all point clouds within the n neighborhood as the reference ground height HR of the grid; n is a positive integer;

[0016] Step 4: Obstacle detection based on height difference: Based on the reference ground height HR of each grid, calculate the height difference deltaH between the point cloud height projected to the grid position and the reference ground height. If deltaH> obstacle height difference TH, it is marked as an obstacle point cloud, otherwise, it is marked as a non-obstacle point cloud;

[0017] Step 5: Obstacle clustering: Set the spatial distance threshold of obstacle clustering to R, perform cluster analysis on the obstacle point cloud, and form obstacle clusters;

[0018] Step 6: Detect the outer contour of the obstacle mass to obtain the minimum enveloping cuboid Rmin of the obstacle mass;

[0019] Step 7: Calculate the left and right boundaries of each Rmin in the horizontal direction and the boundary in the height direction, and extract the rectangular area formed by the left and right boundaries and the height boundary;

[0020] Step 8: Using the joint calibration results of the 3D LiDAR and color camera, transform the extracted rectangular area into the image coordinate system and superimpose it on the video image.

[0021] Preferably, the traversable area detection is based on the 3D point cloud information of the 3D laser radar, and the traversable area detection result is superimposed on the video image through the joint calibration of the 3D laser radar and the color camera. The specific process includes:

[0022] Step 1: According to the process of "positive obstacle detection based on 3D point cloud information of 3D laser radar, and superimposing the positive obstacle detection results on the video image through joint calibration of 3D laser radar and color camera", the obstacle point cloud is projected onto the 2D grid map;

[0023] Step 2: Taking the projection position of the vehicle body coordinates on the two-dimensional grid map as the origin, within the 180-degree range from the left side to the right side of the vehicle body, with a horizontal angle resolution of interval t degrees, use the Breshman straight line algorithm to calculate the maximum distance of the line segment that can be extended in each angular direction of the passable area, and record the end point of the line segment;

[0024] Step 3: Record the coordinates of the end of the passable area that can be reached at each angle from left to right to form a set C;

[0025] Step 4: Smooth the area enclosed by set C to obtain set C2;

[0026] Step 5: Use the joint calibration results of the 3D lidar and color camera to superimpose C2 on the video image.

[0027] Preferably, the negative obstacle detection is performed based on the three-dimensional point cloud information of the three-dimensional laser radar, and the negative obstacle detection result is superimposed on the video image through the joint calibration of the three-dimensional laser radar and the color camera. The specific process includes:

[0028] Step 1: Establish a 3D laser radar scanning model based on the scanning angle distribution of the 3D laser radar;

[0029] Step 2: Calculate the horizontal spacing of point clouds on adjacent radar lines at the same horizontal angle and compare the height values;

[0030] Step 3: The radar point clouds that meet the requirements of horizontal spacing between point clouds on adjacent radar lines and the height constraints are recorded as point clouds of interest;

[0031] Step 4: Cluster the point cloud of interest using a density-based noise spatial clustering method to form clusters;

[0032] Step 5: For the point clouds inside the cluster, select the point clouds distributed on the same radar line as representatives, and calculate the horizontal horizontal spacing of the point clouds on the radar line. If the horizontal horizontal spacing of the point clouds meets the preset distance constraint, it indicates that the cluster is a valid negative obstacle detection, which is defined as Bvalid.

[0033] Step 6: Negative obstacle description: Use the left and right boundaries of the point cloud in Bvalid as the boundaries for describing negative obstacles; use the longest longitudinal distance of the point cloud in Bvalid as the farthest boundary of the negative obstacle, and find the shortest distance of the point cloud on the L+1 radar line corresponding to the point cloud in Bvalid as the shortest boundary of the negative obstacle;

[0034] Step 7: Using the joint calibration results of the 3D LiDAR and color camera, superimpose the negative obstacle detection results onto the video image.

[0035] Preferably, dynamic obstacles such as pedestrians and vehicles are identified and marked on the image using deep learning-based model training. The specific process includes:

[0036] Step 1: Collect image information containing pedestrians and vehicles;

[0037] Step 2: Label samples of pedestrians and vehicles;

[0038] Step 3: Based on the convolutional neural network architecture, use deep neural network to train pedestrian and vehicle models;

[0039] Step 4: Obtain image information from the color camera;

[0040] Step 5: Use the trained model to identify the target;

[0041] Step 6: Set the target recognition rate threshold, record the targets that meet the threshold requirement as valid detection targets, and mark them on the image.

[0042] Preferably, 50 meters is set as the maximum distance of the line segment.

[0043] Preferably, the vehicle-mounted system also includes a vehicle-mounted industrial computer for collecting data and result output from three-dimensional laser radar and color camera, and superimposing the results and publishing them through a wireless link.

[0044] The present invention also provides a vehicle-mounted system used in the process of implementing the method.

[0045] The present invention also provides an application of the method in the technical field of assisted remote control driving.

[0046] The present invention also provides an application of the method in the technical field of remote control of ground unmanned systems.

[0047] (III) Beneficial effects

[0048] The present invention provides a situation awareness method and a vehicle-mounted system for assisted remote control driving of a ground unmanned platform. By installing an environmental perception sensor on the unmanned platform, situation awareness such as traversable area detection and dangerous object identification are performed on the remote driving environment of the unmanned platform, and the situation awareness information is superimposed on the video image to assist remote control driving, improve the safety of remote control driving, reduce driving fatigue of operators, enhance the friendliness of human-computer interaction, and improve the driving experience of remote control of the unmanned platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a principle block diagram of the ground unmanned platform assisted remote control driving situation awareness vehicle-mounted system of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, content, and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings and examples.

[0051] The present invention provides a situation awareness method and a vehicle-mounted system for assisted remote control driving of a ground unmanned platform. The method uses the vehicle-mounted sensors of the ground unmanned platform to perceive the environment, detect and superimpose images of traversable areas, positive and negative static obstacles, dynamic obstacles, etc., provide situation awareness information for the remote control driver, realize safety warning of the driving environment, and improve the decision-making efficiency of the remote control driver.

[0052] To achieve the above invention objectives, the solution of the present invention needs to realize the detection of dangerous factors and passable areas in the driving environment, including positive and negative obstacle detection, dynamic obstacle detection such as pedestrians and vehicles, passable area detection, etc., and at the same time superimpose the above situation awareness information on the video image. Figure 1, the relevant hardware equipment of the vehicle-mounted system includes: 3D laser radar, color camera and vehicle-mounted industrial computer, etc. Among them, the 3D laser radar is used to obtain the 3D point cloud information around the driving environment of the unmanned platform, perform feature extraction and data modeling, and detect positive and negative obstacles and passable areas; through the joint calibration of the 3D laser radar and the color camera, the detection results are superimposed on the video image to provide the remote control driver with auxiliary information such as obstacle warning information and passable areas. The color camera is used to obtain the visual information of the driving environment of the unmanned platform, and the dynamic obstacles such as pedestrians and vehicles are identified and marked on the image through the model training based on deep learning; the vehicle-mounted industrial computer is responsible for sensor data acquisition, data collection and result output, and the superimposed results are published through the wireless link.

[0053] Among them, the positive obstacle detection is based on the 3D point cloud information of the 3D laser radar, and the positive obstacle detection results are superimposed on the video image through the joint calibration of the 3D laser radar and the color camera. The specific process includes:

[0054] Step 1: Obtain 3D radar point cloud data of the remote driving environment of the ground unmanned platform;

[0055] Step 2: Rasterize the 3D radar point cloud data, project the 3D radar point cloud data onto a 2D overhead grid map, and count the number of point clouds and height information in each grid;

[0056] Step 3: Set the ground plane reference, set the n-neighborhood of each grid as the reference basis for the ground height, and take the lowest height of all point clouds within the n-neighborhood as the reference ground height HR of the grid;

[0057] Step 4: Obstacle detection based on height difference. Based on the reference ground height HR of each grid, calculate the height difference deltaH between the point cloud height projected to the grid position and the reference ground height. If deltaH> obstacle height difference TH, it is marked as an obstacle point cloud, otherwise, it is marked as a non-obstacle point cloud;

[0058] Step 5: Obstacle clustering. Set the spatial distance threshold of obstacle clustering to R, perform cluster analysis on the obstacle point cloud, and form obstacle clusters;

[0059] Step 6: Detect the outer contour of the obstacle mass to obtain the minimum enveloping cuboid Rmin of the obstacle mass;

[0060] Step 7: Calculate the left and right boundaries of each Rmin in the horizontal direction and the boundary in the height direction, and extract the rectangular area formed by the left and right boundaries and the height boundary;

[0061] Step 8: Using the joint calibration results of the 3D LiDAR and color camera, transform the extracted rectangular area into the image coordinate system and superimpose it on the video image.

[0062] The traversable area detection is based on the 3D point cloud information of the 3D laser radar, and the traversable area detection results are superimposed on the video image through the joint calibration of the 3D laser radar and the color camera. The specific process includes:

[0063] Step 1: According to the above positive obstacle detection process, the obstacle point cloud is projected onto a two-dimensional grid map to obtain an obstacle / passable binary map;

[0064] Step 2: Taking the projection position of the vehicle body coordinates on the two-dimensional grid map as the origin, within the 180-degree range from the left to the right of the vehicle body, with a horizontal angle resolution of interval t degrees, use the Breshman straight line algorithm to calculate the maximum distance that the passable area can extend in each angle direction (50 meters is the maximum distance of the line segment), and record the end point of the line segment;

[0065] Step 3: Record the coordinates of the end of the passable area that can be reached at each angle from left to right to form a set C;

[0066] Step 4: Smooth the area enclosed by set C to obtain set C2;

[0067] Step 5: Use the joint calibration results of the 3D lidar and color camera to superimpose C2 on the video image.

[0068] Negative obstacle detection based on 3D point cloud information of 3D laser radar, and superimposing the negative obstacle detection results on the video image through joint calibration of 3D laser radar and color camera. The specific process includes:

[0069] Step 1: Establish a 3D laser radar scanning model based on the scanning angle distribution of the 3D laser radar;

[0070] Step 2: As shown in formula (4), calculate the horizontal spacing of point clouds on adjacent radar lines at the same horizontal angle and compare the height values ​​at the same time;

[0071] Step 3: For the radar point cloud that satisfies both formula (5) and formula (6), it is recorded as the point cloud of interest;

[0072] Step 4: Cluster the point cloud of interest using the density-based noise spatial clustering method (DBSCAN) to form clusters;

[0073] Step 5: For the point clouds inside the cluster, select the point clouds distributed on the same radar line as representatives, and calculate the average spacing of the point clouds on the radar line. If formula (9) is satisfied, it indicates that the cluster is a valid negative obstacle detection, which is defined as Bvalid.

[0074] Step 6: Negative obstacle description. Use the left and right boundaries of the point cloud in Bvalid as the boundaries for describing negative obstacles; use the longest longitudinal distance of the point cloud in Bvalid as the farthest boundary of the negative obstacle, and find the shortest distance of the point cloud on the L+1 radar line corresponding to the point cloud in Bvalid as the shortest boundary of the negative obstacle;

[0075] Step 7: Using the joint calibration results of the 3D LiDAR and color camera, superimpose the negative obstacle detection results onto the video image.

[0076] Using deep learning-based model training, dynamic obstacles such as pedestrians and vehicles are identified and annotated on the image. The specific process includes:

[0077] Step 1: Collect image information containing pedestrians and vehicles;

[0078] Step 2: Label samples of pedestrians and vehicles;

[0079] Step 3: Based on the convolutional neural network architecture, use deep neural network to train pedestrian and vehicle models;

[0080] Step 4: Obtain image information from the color camera;

[0081] Step 5: Use the trained model to identify the target;

[0082] Step 6: Set the target recognition rate threshold, record the targets that meet the threshold requirement as valid detection targets, and mark them on the image.

[0083] It can be seen that the situational awareness method and vehicle-mounted system for assisted remote control driving of ground unmanned platforms provided by the present invention can assist remote driving, provide the driver with safety warnings of positive and negative obstacles, dynamic obstacles such as pedestrians and vehicles, and describe situational information such as safe and passable areas. The invention can effectively reduce the workload of remote control personnel, especially reduce visual fatigue, improve the safety of remote control, decision-making efficiency and accuracy, and at the same time improve the intelligence level of remote control.

[0084] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A situation awareness method for assisted remote control driving of a ground unmanned platform, characterized in that: The method is implemented by using a vehicle-mounted system, which includes a three-dimensional laser radar and a color camera. In the method, the three-dimensional laser radar is used to obtain three-dimensional point cloud information around the driving environment of the unmanned platform, and positive and negative obstacles and passable areas are detected; Through the joint calibration of 3D laser radar and color camera, the detection results are superimposed on the video image to provide the remote control driver with auxiliary information such as obstacle warning information and passable area. The color camera is also used to obtain visual information of the unmanned platform's driving environment, and the two dynamic obstacles, pedestrians and vehicles, are identified and marked on the image through model training based on deep learning. Among them, the positive obstacle detection is based on the 3D point cloud information of the 3D laser radar, and the positive obstacle detection results are superimposed on the video image through the joint calibration of the 3D laser radar and the color camera. The specific process includes: Step 1: Obtain 3D radar point cloud data of the remote driving environment of the ground unmanned platform; Step 2: Rasterize the 3D radar point cloud data, project the 3D radar point cloud data onto a 2D overhead grid map, and count the number of point clouds and height information in each grid; Step 3: Set the n neighborhood of each grid as the reference basis for the ground height, and take the lowest height of all point clouds within the n neighborhood as the reference ground height HR of the grid; n is a positive integer; Step 4: Obstacle detection based on height difference: Based on the reference ground height HR of each grid, calculate the height difference deltaH between the point cloud height projected to the grid position and the reference ground height. If deltaH> obstacle height difference TH, it is marked as an obstacle point cloud, otherwise, it is marked as a non-obstacle point cloud; Step 5: Obstacle clustering: Set the spatial distance threshold of obstacle clustering to R, perform cluster analysis on the obstacle point cloud, and form obstacle clusters; Step 6: Detect the outer contour of the obstacle mass to obtain the minimum enveloping cuboid Rmin of the obstacle mass; Step 7: Calculate the left and right boundaries of each Rmin in the horizontal direction and the boundary in the height direction, and extract the rectangular area formed by the left and right boundaries and the height boundary; Step 8: Using the joint calibration results of the 3D laser radar and the color camera, transform the extracted rectangular area into the image coordinate system and superimpose it on the video image; The traversable area detection is based on the 3D point cloud information of the 3D laser radar, and the traversable area detection results are superimposed on the video image through the joint calibration of the 3D laser radar and the color camera. The specific process includes: Step 1: According to the process of "positive obstacle detection based on 3D point cloud information of 3D laser radar, and superimposing the positive obstacle detection results on the video image through joint calibration of 3D laser radar and color camera", the obstacle point cloud is projected onto the 2D grid map; Step 2: Taking the projection position of the vehicle body coordinates on the two-dimensional grid map as the origin, within the 180-degree range from the left side to the right side of the vehicle body, with a horizontal angle resolution of interval t degrees, use the Breshman straight line algorithm to calculate the maximum distance of the line segment that can be extended in each angular direction of the passable area, and record the end point of the line segment; Step 3: Record the coordinates of the end of the passable area that can be reached at each angle from left to right to form a set C; Step 4: Smooth the area enclosed by set C to obtain set C2; Step 5: Using the joint calibration results of the 3D laser radar and the color camera, C2 is superimposed on the video image; Negative obstacle detection based on 3D point cloud information of 3D laser radar, and superimposing the negative obstacle detection results on the video image through joint calibration of 3D laser radar and color camera. The specific process includes: Step 1: Establish a 3D laser radar scanning model based on the scanning angle distribution of the 3D laser radar; Step 2: Calculate the horizontal spacing of point clouds on adjacent radar lines at the same horizontal angle and compare the height values; Step 3: The radar point clouds that meet the requirements of horizontal spacing between point clouds on adjacent radar lines and the height constraints are recorded as point clouds of interest; Step 4: Cluster the point cloud of interest using a density-based noise spatial clustering method to form clusters; Step 5: For the point clouds inside the cluster, select the point clouds distributed on the same radar line as representatives, and calculate the horizontal horizontal spacing of the point clouds on the radar line. If the horizontal horizontal spacing of the point clouds meets the preset distance constraint, it indicates that the cluster is a valid negative obstacle detection, which is defined as Bvalid. Step 6: Negative obstacle description: Use the left and right boundaries of the point cloud in Bvalid as the boundaries for describing negative obstacles; use the longest longitudinal distance of the point cloud in Bvalid as the farthest boundary of the negative obstacle, and find the shortest distance of the point cloud on the L+1 radar line corresponding to the point cloud in Bvalid as the shortest boundary of the negative obstacle; Step 7: Using the joint calibration results of the 3D LiDAR and color camera, superimpose the negative obstacle detection results onto the video image.

2. The method according to claim 1, characterized in that Using deep learning-based model training, pedestrians and vehicles, two dynamic obstacles, are identified and annotated on the image. The specific process includes: Step 1: Collect image information containing pedestrians and vehicles; Step 2: Label samples of pedestrians and vehicles; Step 3: Based on the convolutional neural network architecture, use deep neural network to train pedestrian and vehicle models; Step 4: Obtain image information from the color camera; Step 5: Use the trained model to identify the target; Step 6: Set the target recognition rate threshold, record the targets that meet the threshold requirement as valid detection targets, and mark them on the image.

3. The method according to claim 1, characterized in that Assume 50 meters as the maximum distance of the line segment.

4. The method according to claim 1, characterized in that The vehicle-mounted system also includes a vehicle-mounted industrial computer, which is used to collect data and output results from three-dimensional laser radar and color cameras, and to superimpose the results and publish them via a wireless link.

5. Application of the method according to any one of claims 1 to 4 in the field of assisted remote control driving technology.

6. Application of the method as claimed in any one of claims 1 to 4 in the field of remote control technology of ground unmanned systems.

Images