Remote obstacle detection method based on image and radar fusion

Abstract

The invention belongs to the technical field of urban rail train operation protection, and discloses a remote obstacle detection method based on image and radar fusion. According to the invention, the scene and scheme of urban rail train obstacle detection are analyzed; determining the distance requirement of the urban rail train for detecting the obstacle and the obstacle size requirement matched with the distance; key data required by obstacle detection and constraint conditions of the urban rail train running speed are analyzed; determining the highest running speed of the train during obstacle detection by using data acquired by a laser radar, an IMU (Inertial Measurement Unit) and a camera and detection; according to the highest running speed, train braking response time and the longest braking distance are analyzed; due to the fact that a laser radar sensor has distance limitation, an image estimation target distance algorithm and an image and radar fusion algorithm are determined; according to a fusion algorithm, detection of obstacles which are 200 m away from the front of the train and are larger than 1.5 m in height is compensated, and the requirement for long-distance obstacle detection in a train operation scene is met.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of urban rail train operation protection, and in particular to a long-distance obstacle detection method based on image and radar fusion. Background Art

[0002] The obstacle detection system of urban rail trains provides accurate train position information and obstacle information ahead for the train automatic control system and automatic operation system, which is an important prerequisite for realizing automatic driving of urban rail trains. With the development of unmanned driving technology, higher requirements are put forward for the detection range, accuracy and safety of obstacle detection systems.

[0003] At present, most train obstacle detection systems are based on LiDAR and IMU (inertial sensor). When the IMU data (inertial navigation data) is offset greatly or the LiDAR cannot detect distant targets, the obstacle detection function is unavailable, and the train loses its position or enters an unsafe detection state. Therefore, it is very necessary to use other sensor data to supplement the information.

[0004] Among the currently used laser radar and image fusion methods, decision fusion is the main method, which only focuses on obstacle detection results and ignores the distance information of obstacles. When the laser radar cannot provide the location information of obstacles, the image information is used to supplement the location information of obstacles, which brings great challenges to the accuracy and safety of the obstacle detection system.

[0005] Designing a more effective information fusion algorithm to achieve long-distance obstacle detection is an effective way to improve the detection range, accuracy and safety of the obstacle detection system. Summary of the invention

[0006] The purpose of the present invention is to provide a long-distance obstacle detection method based on image and radar fusion. In view of the problem that the prior art ignores the distance information of obstacles, the present invention detects long-distance obstacles not detected by the laser radar and compensates for the distance information, thereby improving the detection range, accuracy and safety of the obstacle detection system.

[0007] To achieve the above object, the present invention provides a long-distance obstacle detection method based on image and radar fusion, comprising the following steps: Step 1: The camera generates image data, the laser radar generates point cloud data, and the inertial sensor generates inertial navigation data. The point cloud data includes ground point cloud data, target point cloud data, and non-target point cloud data. The image data, point cloud data, and inertial navigation data are time-aligned, and the image data, point cloud data, and inertial navigation data are transformed into coordinate systems to achieve spatial coordinate alignment. Step 2: Position estimation based on the fusion of inertial navigation data and point cloud data, combined with high-precision maps to identify targets in the area; Step 3: Detect and estimate the distance information of the target based on the image, and judge the radar detection result based on the spatial coordinate position in the corresponding world coordinate system. When the radar detection result is valid, output the radar detection result; Step 4: Based on the discrimination results of image detection and radar detection, when there is only image detection result, the nearest related point search is performed in combination with the unfiltered ground point cloud data, and the target distance information estimated by the unfiltered ground point cloud data is output; Step 5: Based on the distance information of the nearest related point, the distance data of the nearest related point of the current frame is combined with the distance data of the previous 4 frames to find the average value to obtain the final target detection result and distance information.

[0008] Furthermore, in step 1, the specific process of performing time alignment and spatial coordinate alignment of the image data, point cloud data, and inertial navigation data includes: S1: Compare the timestamps of image data, point cloud data and inertial navigation data based on the nearest time correlation method, taking the timestamp of point cloud data as the benchmark; S2: Compare whether the timestamps of the image data in the frame before and after the timestamp of the point cloud data are the same or within a very small range, and select the frame with the smallest timestamp difference as the aligned image data; S3: Compare the timestamps of the inertial navigation data in the frame before and after the timestamp of the point cloud data to see if they are the same or within a very small range. Under the assumption that the data of the previous and next frames of the inertial navigation data change in a fixed increment, estimate the inertial navigation data with the same timestamp as the point cloud data for alignment. S4: Based on the rotation and translation relationship of the base points of the camera, lidar and inertial sensor coordinate systems, the projection transformation method is used to calculate the transformation matrix between the coordinate systems to ensure that the origins of the camera, lidar and inertial sensor coordinate systems are in the same coordinate system.

[0009] Furthermore, in step 2, the process of pose estimation based on the fusion of inertial navigation data and point cloud data and identifying targets in the area in combination with high-precision maps is as follows: based on the pre-established high-precision map of the route, the pose of the train is initialized using the landmark positioning method, the position of the train in the map is calculated in combination with the inertial navigation data, the ground point cloud data and non-target point cloud data collected by the lidar are eliminated, and the real obstacle targets are identified.

[0010] Furthermore, in step 3, the image detects and estimates the distance information of the target using an image estimation target distance algorithm, and estimates the center distance for different types of targets respectively. The specific steps are as follows: S5: performing distortion correction based on the image data collected by the camera, obtaining the internal and external parameter matrix of the camera, and setting the distance offset of the image data; S6: Detect objects based on image data, distinguish object types, and assign different pixel widths and pixel heights to different types of objects; S7: Calculate the positions of different types of targets in width and height and then superimpose the distance offsets, compare the minimum values ​​and obtain the estimated distance of the target; The specific processing process based on image detection and radar detection result judgment includes: based on the image estimation target distance algorithm to detect the target type and distance, calculate whether there is a target in the corresponding world coordinate system position of all point cloud data according to the transformation matrix, if there is a radar detection result at this position, output the radar detection result, otherwise, jump to step 4.

[0011] Furthermore, in step 4, the specific processing process of obtaining the estimated target distance information based on the image detection result and the characteristics of the ground point cloud data includes: Based on the image estimated distance and radar detection results, if there is only an image detection result at the same position in the same world coordinate system, the ground point cloud data is compared with the image estimated distance, that is, .

[0012] Furthermore, in step 5, based on the distance information of the relevant points, the specific processing process of calculating the final target detection result and the distance information includes: Based on the estimated distance information of the ground point cloud data and the image detection results, the results of the first four frames are averaged with the current frame, that is, .

[0013] Furthermore, the method further includes step 6: outputting a security protection solution. The process of outputting a security protection solution includes the following steps: S8: Fuse the image data, point cloud data and inertial navigation data, add the train running speed constraint condition, and determine the validity of the obstacle detection result; S9: Based on the effectiveness of the obstacle detection results, combined with the urban rail train obstacle detection scene and type, the dangerous obstacle type is identified; S10: Based on the analysis of the dangerous obstacle type, determine whether the obstacle distance is a safe distance and output a safety protection plan.

[0014] Furthermore, in step S5, the specific processing flow of performing distortion correction on the image data includes: A special calibration plate is used to take photos, and the world coordinate system is converted to the camera coordinate system using corner point detection, that is, ; The radial distortion coefficient and tangential distortion coefficient are calculated by Taylor formula expansion and the internal parameter matrix is ​​obtained by reconstructing the contrast error. Set the distance offset based on the values ​​of the internal and external parameter matrices; In step S6, the specific process of performing target detection on the image data and assigning different pixel widths and pixel heights to different types of targets includes: using a deep learning network model to detect the target and obtain the type name, and assigning pixel width and height based on prior information to a specific type; In step S7, the specific processing flow of estimating the target distance from the image data includes: Multiply the pixel width and height of the target detected by the deep learning network model by the focal length, divide them by the set pixel width and height respectively, and add the distance offset. Compare the minimum values ​​of the width position and the height position, and use the minimum value as the estimated distance of the target.

[0015] Furthermore, in step S8, based on the fusion of image data, point cloud data and inertial navigation data, the specific processing process of determining the validity of the obstacle detection result includes: based on the image data, point cloud data and inertial navigation data, the type and distance information of the obstacle are obtained through the image estimation target distance algorithm and the image and radar fusion algorithm, and according to the constraint condition of the train running speed, the result exceeding the running speed is invalid; In step S9, based on the validity of the obstacle detection result, the specific process of determining the type of dangerous obstacle includes: Analyze the effectiveness of obstacles based on the train speed constraints, and analyze whether the detected obstacles are dangerous obstacles based on the obstacle detection scenarios and types of urban rail trains, and output the information of dangerous obstacles; In step S10, based on the validity of the obstacle detection result, the specific processing process of determining the type of dangerous obstacle includes: based on the determination information of the dangerous obstacle, comparing whether the distance of the obstacle is within the safe distance. If it is within the safe distance, only voice prompt is given; otherwise, an alarm is given and a reminder is given whether to brake.

[0016] Beneficial effects: Beneficial effects: The present invention provides a long-distance obstacle detection method based on image and radar fusion, which takes into account common scenarios of obstacle detection by urban rail trains, clarifies the distance requirements for obstacle detection by urban rail trains and the obstacle size requirements that match the distance, takes into account the constraints of the running speed of urban rail trains, further constrains the distance range required for obstacle detection, takes into account the problem that the existing image and radar decision fusion does not compensate for distance information, and proposes a distance estimation method for the nearest target point of fused image detection and laser radar, which is not only conducive to expanding the obstacle detection range, but also conducive to improving the accuracy of obstacle detection; and protects the obstacle detection results of image and radar fusion, which is conducive to improving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of a laser radar detecting a target according to an embodiment of the present invention; Figure 2 is a flow chart of an algorithm for estimating target distance from an image according to an embodiment of the present invention; Figure 3 is a flow chart of an image and radar fusion algorithm involved in an embodiment of the present invention; Figure 4 It is a flow chart of the security protection scheme involved in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The preferred mechanism and implementation method of the present invention are further described below in conjunction with the accompanying drawings and specific implementation methods.

[0019] like Figures 1 to 4 As shown, an embodiment of the present invention discloses a technical solution for a long-distance obstacle detection method based on image and radar fusion.

[0020] The camera generates image data, the lidar generates point cloud data, and the inertial sensor (IMU) generates inertial navigation data (IMU data); Point cloud data includes ground point cloud data, target point cloud data, and non-target point cloud data; The target point cloud data refers to the point cloud data generated by the laser radar within the track width range and above the track rail height; Ground point cloud data refers to the point cloud data generated by the laser radar within the track width and below the height of the track rails; Non-target point cloud data refers to the point cloud data generated by the lidar outside the track width.

[0021] Example 1 The principle of laser radar detection target is as follows Figure 1As shown in the figure, the laser radar obtains the distance information of each received point by calculating the time deviation between the transmitted light beam and the received light beam; as the distance to the target increases, the intensity of the light beam that can be reflected by the target decreases, and the number of target point clouds gradually decreases. Therefore, smaller targets themselves reflect fewer point clouds, which disappear as the distance increases. At the same time, in the dark or when the light is too strong, the reflectivity of the target will be greatly affected, thereby affecting the number of point clouds of the target. Therefore, the detection distance of the laser radar for targets of different sizes is different, and the detection distance of the same target in different environments is also different.

[0022] A long-distance obstacle detection method based on image and radar fusion is Figure 3 As shown, the following steps are included: Step 1: The camera generates image data, the laser radar generates point cloud data, and the inertial sensor generates inertial navigation data. The point cloud data includes ground point cloud data, target point cloud data, and non-target point cloud data. The image data, point cloud data, and inertial navigation data are time-aligned, and the image data, point cloud data, and inertial navigation data are transformed into coordinate systems to achieve spatial coordinate alignment. Step 2: Position estimation based on the fusion of inertial navigation data and point cloud data, combined with high-precision maps to identify targets in the area; Step 3: Detect and estimate the distance information of the target based on the image, and judge the radar detection result based on the spatial coordinate position in the corresponding world coordinate system. When the radar detection result is valid, output the radar detection result; Step 4: Based on the discrimination results of image detection and radar detection, when there is only image detection result, the nearest related point search is performed in combination with the unfiltered ground point cloud data, and the target distance information estimated by the unfiltered ground point cloud data is output; Step 5: Based on the distance information of the nearest related point, the distance data of the nearest related point of the current frame is combined with the distance data of the previous 4 frames to find the average value to obtain the final target detection result and distance information.

[0023] In step 1, the specific process of aligning the image data, point cloud data and IMU data in time and space coordinates includes: S1: Compare the timestamps of image data, point cloud data and inertial navigation data based on the nearest time correlation method, taking the timestamp of point cloud data as the benchmark; S2: Compare whether the timestamps of the image data in the frame before and after the timestamp of the point cloud data are the same or within a very small range, and select the frame with the smallest timestamp difference as the aligned image data; S3: Compare the timestamps of the inertial navigation data in the frame before and after the timestamp of the point cloud data to see if they are the same or within a very small range. Under the assumption that the data of the previous and next frames of the inertial navigation data change in a fixed increment, estimate the inertial navigation data with the same timestamp as the point cloud data for alignment. S4: Based on the rotation and translation relationship of the base points of the camera, lidar and inertial sensor coordinate systems, the projection transformation method is used to calculate the transformation matrix between the coordinate systems to ensure that the origins of the camera, lidar and inertial sensor coordinate systems are in the same coordinate system.

[0024] In step 2, the process of pose estimation based on the fusion of inertial navigation data and point cloud data and identifying targets in the area in combination with high-precision maps is as follows: based on the pre-established high-precision map of the route, the pose of the train is initialized using the landmark positioning method, the position of the train in the map is calculated in combination with the inertial navigation data, the ground point cloud data and non-target point cloud data collected by the lidar are eliminated, and the real obstacle targets are identified.

[0025] In step 3, the image is detected and the distance information of the target is estimated using the image estimation target distance algorithm, such as Figure 2 As shown, the center distance is estimated for different types of targets respectively. The specific steps are as follows: S5: performing distortion correction based on the image data collected by the camera, obtaining the internal and external parameter matrix of the camera, and setting the distance offset of the image data; In step S5, the specific processing flow of performing distortion correction on the image data includes: A special calibration plate is used to take photos, and the world coordinate system is converted to the camera coordinate system using corner point detection, that is, ; The radial distortion coefficient and tangential distortion coefficient are calculated by Taylor formula expansion and the internal parameter matrix is ​​obtained by reconstructing the contrast error. Sets the distance offset based on the values ​​of the inner and outer parameter matrices.

[0026] S6: Detect objects based on image data, distinguish object types, and assign different pixel widths and pixel heights to different types of objects; In step S6, target detection is performed on the image data, and the specific processing process of assigning different pixel widths and pixel heights to different types of targets includes: using a deep learning network model to detect the target and obtain the type name, and assigning pixel width and height based on prior information to a specific type.

[0027] S7: Calculate the positions of different types of targets in width and height and then superimpose the distance offsets, compare the minimum values ​​and obtain the estimated distance of the target; In step S7, the specific processing flow of estimating the target distance from the image data includes: Multiply the pixel width and height of the target detected by the deep learning network model by the focal length, divide them by the set pixel width and height respectively, and add the distance offset, compare the minimum value of the width position and the height position, and use the minimum value as the estimated distance of the target; The specific processing process based on image detection and radar detection result judgment includes: based on the image estimation target distance algorithm to detect the target type and distance, calculate whether there is a target in the corresponding world coordinate system position of all point cloud data according to the transformation matrix, if there is a radar detection result at this position, output the radar detection result, otherwise, jump to step 4.

[0028] In step 4, the specific processing process of obtaining the estimated target distance information based on the image detection results and the characteristics of the ground point cloud data includes: Based on the image estimated distance and radar detection results, if there is only an image detection result at the same position in the same world coordinate system, the ground point cloud data is compared with the image estimated distance, that is, ,in, Indicates the estimated distance of distant obstacles in the current frame, Indicates the distance range beyond the laser radar limit. Indicates the estimated distance depth information of the current frame image, and outputs distance information with more points .

[0029] In step 5, based on the distance information of the relevant points, the specific process of calculating the final target detection result and distance information includes: Based on the estimated distance information of the ground point cloud data and the image detection results, the results of the first four frames are averaged with the current frame, that is, , Represents the output distance of the obstacle detected by the k-th frame point cloud data, Indicates the output distance of the obstacle detection of the point cloud data of the first i frames of the kth frame, and the final output is the mean distance The distance to detect the target.

[0030] Example 2 The steps of this embodiment are basically the same as those of embodiment 1, except that this embodiment further includes step 6: outputting a security protection scheme. The process of outputting a security protection scheme includes the following steps: S8: Fuse the image data, point cloud data and inertial navigation data, add the train running speed constraint condition, and determine the validity of the obstacle detection result; S9: Based on the effectiveness of the obstacle detection results, combined with the urban rail train obstacle detection scene and type, the dangerous obstacle type is identified; S10: Based on the analysis of the dangerous obstacle type, determine whether the obstacle distance is a safe distance and output a safety protection plan.

[0031] In step S8, based on the fusion of image data, point cloud data and inertial navigation data, the specific processing process of determining the validity of the obstacle detection result includes: based on the image data, point cloud data and inertial navigation data, the type and distance information of the obstacle are obtained through the image estimation target distance algorithm and the image and radar fusion algorithm, and according to the constraint condition of the train running speed, the result exceeding the running speed is invalid; In step S9, based on the validity of the obstacle detection result, the specific process of determining the type of dangerous obstacle includes: Analyze the effectiveness of obstacles based on the train speed constraints, and analyze whether the detected obstacles are dangerous obstacles based on the obstacle detection scenarios and types of urban rail trains, and output the information of dangerous obstacles; In step S10, based on the validity of the obstacle detection result, the specific processing process of determining the type of dangerous obstacle includes: based on the determination information of the dangerous obstacle, comparing whether the distance of the obstacle is within the safe distance. If it is within the safe distance, only voice prompt is given; otherwise, an alarm is given and a reminder is given whether to brake.

[0032] Example 3 The steps of this embodiment are basically the same as those of Embodiment 2, except that the technical solution of this embodiment is as follows: analyzing the scenarios and solutions of obstacle detection by urban rail trains; determining the distance requirement for obstacle detection by urban rail trains and the obstacle size requirement matching the distance; analyzing the key data required for obstacle detection and the constraints of the running speed of urban rail trains according to the distance requirement and the obstacle size requirement; confirming the use of data collected by laser radar, IMU and camera for obstacle detection and the maximum running speed of the train during detection, and analyzing the train braking response time and the longest braking distance according to the maximum running speed; due to the distance limitation problem of laser radar sensors, determining an image estimation target distance algorithm and an image and radar fusion algorithm, and according to the fusion algorithm, compensating for obstacle detection 200m ahead of the train and with a height greater than 1.5m, meeting the long-distance obstacle detection requirements in train running scenarios, and providing a safety protection solution for train running safety.

[0033] The scenarios and solutions for obstacle detection in urban rail trains include: listing the environmental conditions and obstacle types in the train operation scenarios according to the requirements of the obstacle detection function, analyzing the distance restrictions of sensors such as lidar, camera, IMU, etc. to different obstacles under different environmental conditions, using Gigabit Ethernet to transmit huge lidar point cloud data, and combining IMU data to correct the train's position in the map to achieve train positioning. At the same time, when the lidar has no data output or is unavailable, the image is used to estimate the depth information of the obstacle to compensate the distance and achieve long-distance obstacle detection.

[0034] The distance requirement for urban rail trains to detect obstacles and the obstacle size requirement that matches the distance include: analyzing the distance required for urban rail trains to detect obstacles based on the average operating speed and safety requirements of urban rail trains in various operating scenarios, adjusting the obstacle size according to different distances based on the actual imaging principle of "near is big and far is small", and obtaining the obstacle size requirement that matches the distance.

[0035] The "key data" in the key data required for obstacle detection are analyzed, including: point cloud data collected by radar, inertial navigation IMU data and image data. Considering that a single lidar point cloud data can only detect targets but cannot distinguish obstacles, after establishing a point cloud map, the IMU data is used to correct the train's position in the map to determine whether the point cloud target is an obstacle in the track area. Lidar has different distance limitations in different environments. Considering the universality of the obstacle detection range, image data can compensate for the distance of obstacles when radar point cloud data is not available, thereby expanding the obstacle detection range.

[0036] The "constraints on the maximum running speed of the train" in the constraints on the running speed of the urban rail train required for obstacle detection are analyzed, including: according to the constraints of displacement bias and speed bias when the laser radar and IMU are fused to calculate the positioning, the maximum running speed of the train is calculated, that is, ,in Indicates the maximum operating speed of the train. Indicates the maximum allowed speed without resetting the IMU parameters. Represents the maximum displacement bias calculated by the IMU, It represents the interval time between valid data frames. Combined with the maximum allowable speed offset and displacement offset, the minimum value is selected as the constraint condition of the train running speed to meet the running conditions required for the train to detect obstacles ahead.

[0037] The train braking response time and the longest braking distance include: according to the constraints of the train's maximum operating speed and the train braking protection curve, the distance traveled within the driver's reaction and system response time and the distance traveled along the protection curve are analyzed to obtain the longest braking distance. The longest braking distance is combined with the distance requirements to further constrain the distance range required for obstacle detection.

[0038] The image estimation target distance algorithm and the image and radar fusion algorithm include: according to the fixed installation position of the lidar, IMU and camera, the coordinate system projection transformation is used to align the point cloud data, IMU data and image data in the spatial coordinate system, that is, ,in, Represents the horizontal coordinate in the image coordinate system. Represents the vertical coordinate in the image coordinate system. Represents the coordinates perpendicular to the ground in the camera coordinate system. Represents the unit width of a pixel in the image plane, Represents the unit height of the pixel in the image plane, Represents the horizontal coordinate in the pixel coordinate system. Represents the vertical coordinate in the pixel coordinate system. represents the focal length of the camera, represents the rotation matrix, represents the translation matrix, Represents LiDAR x The coordinates in the axis coordinate system, Represents LiDAR y The coordinates in the axis coordinate system, Represents LiDAR z The coordinates in the axis coordinate system are compared by the nearest neighbor correlation time method for time alignment. In the time-aligned point cloud and image data, the image is detected to obtain the target type and estimate the distance information. At the same time, the image result is compared with the radar result. When the image detects an obstacle and the point cloud data does not have a target at the corresponding position, the target position in the image is fitted by tiling the ground point cloud, and the estimated distance of the point cloud is used to correct the estimated distance depth information of the image, that is, ,in, Indicates the estimated distance of distant obstacles in the current frame, Indicates the distance range beyond the laser radar limit. Represents the distance depth information estimated by the current frame image, and then the distance value of the long-distance obstacle is calibrated by taking the average of the distance results of every 5 frames, that is, , Represents the output distance of the obstacle detected by the k-th frame point cloud data, It represents the output distance of the obstacle detected by the point cloud data of the previous i frames of the kth frame, completing the long-distance obstacle detection task.

[0039] Compensation for obstacle detection 200m ahead of the train and with a height greater than 1.5m, including: based on the characteristics of highly dangerous obstacles in front of the train, such as pedestrians and trains running ahead, the height requirements corresponding to key obstacles are obtained, that is, for obstacles with a height greater than 1.5m, the maximum distance for LiDAR to detect pedestrians and other targets is 200m. For targets more than 200m away, the distance depth information is estimated by image, and then combined with ground point cloud information, obstacle detection 200m away and with a height greater than 1.5m is achieved.

[0040] Provide safety protection solutions for train operation safety, including: reading the obstacle detection results of image and radar fusion, combining the train speed constraints and the obstacle detection scene and type determination of urban rail trains. If the obstacle is within the safe distance, a voice prompt will be given; if the obstacle is within the unsafe distance, an alarm will be directly sounded to remind the driver whether to brake.

[0041] The present invention provides a long-distance obstacle detection method based on image and radar fusion, takes into account common scenarios of obstacle detection by urban rail trains, clarifies the distance requirement for obstacle detection by urban rail trains and the obstacle size requirement matching the distance, takes into account the constraint conditions of the running speed of urban rail trains, further constrains the distance range required for obstacle detection, takes into account the problem that the existing image and radar decision fusion does not compensate for distance information, and proposes a distance estimation method for the nearest target point of fused image detection and laser radar, which is not only conducive to expanding the obstacle detection range, but also conducive to improving the accuracy of obstacle detection; and protects the obstacle detection result of image and radar fusion, which is conducive to improving safety.

[0042] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. However, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A long-distance obstacle detection method based on image and radar fusion, characterized in that: The following steps are involved: Step 1: The camera generates image data, the laser radar generates point cloud data, and the inertial sensor generates inertial navigation data. The point cloud data includes ground point cloud data, target point cloud data, and non-target point cloud data. The image data, point cloud data, and inertial navigation data are time-aligned, and the image data, point cloud data, and inertial navigation data are transformed into coordinate systems to achieve spatial coordinate alignment. Step 2: Position estimation based on the fusion of inertial navigation data and point cloud data, combined with high-precision maps to identify targets in the area; Step 3: Detect and estimate the distance information of the target based on the image, and judge the radar detection result based on the spatial coordinate position in the corresponding world coordinate system. When the radar detection result is valid, output the radar detection result; Step 4: Based on the discrimination results of image detection and radar detection, when there is only image detection result, the nearest related point search is performed in combination with the unfiltered ground point cloud data, and the target distance information estimated by the unfiltered ground point cloud data is output; Step 5: Based on the distance information of the nearest related point, the distance data of the nearest related point of the current frame is combined with the distance data of the previous 4 frames to find the average value to obtain the final target detection result and distance information.

2. The long-distance obstacle detection method based on image and radar fusion according to claim 1, characterized in that: In step 1, the specific process of aligning the image data, point cloud data and inertial navigation data in time and space coordinates includes: S1: Compare the timestamps of image data, point cloud data and inertial navigation data based on the nearest time correlation method, taking the timestamp of point cloud data as the benchmark; S2: Compare whether the timestamps of the image data in the frame before and after the timestamp of the point cloud data are the same or within a very small range, and select the frame with the smallest timestamp difference as the aligned image data; S3: Compare the timestamps of the inertial navigation data in the frame before and after the timestamp of the point cloud data to see if they are the same or within a very small range. Under the assumption that the data of the previous and next frames of the inertial navigation data change in a fixed increment, estimate the inertial navigation data with the same timestamp as the point cloud data for alignment. S4: Based on the rotation and translation relationship of the base points of the camera, lidar and inertial sensor coordinate systems, the projection transformation method is used to calculate the transformation matrix between the coordinate systems to ensure that the origins of the camera, lidar and inertial sensor coordinate systems are in the same coordinate system.

3. The long-distance obstacle detection method based on image and radar fusion according to claim 1, characterized in that: In step 2, the process of pose estimation based on the fusion of inertial navigation data and point cloud data and identifying targets in the area in combination with high-precision maps is as follows: based on the pre-established high-precision map of the route, the pose of the train is initialized using the landmark positioning method, the position of the train in the map is calculated in combination with the inertial navigation data, the ground point cloud data and non-target point cloud data collected by the lidar are eliminated, and the real obstacle targets are identified.

4. The long-distance obstacle detection method based on image and radar fusion according to claim 1, characterized in that: In step 3, the image detects and estimates the distance information of the target using the image estimation target distance algorithm, and estimates the center distance for different types of targets respectively. The specific steps are as follows: S5: performing distortion correction based on the image data collected by the camera, obtaining the internal and external parameter matrix of the camera, and setting the distance offset of the image data; S6: Detect objects based on image data, distinguish object types, and assign different pixel widths and pixel heights to different types of objects; S7: Calculate the positions of different types of targets in width and height and then superimpose the distance offsets, compare the minimum values ​​and obtain the estimated distance of the target; The specific processing process based on image detection and radar detection result judgment includes: based on the image estimation target distance algorithm to detect the target type and distance, calculate whether there is a target in the corresponding world coordinate system position of all point cloud data according to the transformation matrix, if there is a radar detection result at this position, output the radar detection result, otherwise, jump to step 4.

5. The long-distance obstacle detection method based on image and radar fusion according to claim 1, characterized in that: In step 4, the specific processing process of obtaining the estimated target distance information based on the image detection results and the characteristics of the ground point cloud data includes: Based on the image estimated distance and radar detection results, if there is only an image detection result at the same position in the same world coordinate system, the ground point cloud data is compared with the image estimated distance, that is, .

6. The long-distance obstacle detection method based on image and radar fusion according to claim 1, characterized in that: In step 5, based on the distance information of the relevant points, the specific process of calculating the final target detection result and distance information includes: Based on the estimated distance information of the ground point cloud data and the image detection results, the results of the first four frames are averaged with the current frame, that is, 。 7. The long-distance obstacle detection method based on image and radar fusion according to claim 1, characterized in that: The method further includes step 6: outputting a security protection scheme. The process of outputting a security protection scheme includes the following steps: S8: Fuse the image data, point cloud data and inertial navigation data, add the train running speed constraint condition, and determine the validity of the obstacle detection result; S9: Based on the effectiveness of the obstacle detection results, combined with the urban rail train obstacle detection scene and type, the dangerous obstacle type is identified; S10: Based on the analysis of the dangerous obstacle type, determine whether the obstacle distance is a safe distance and output a safety protection plan.

8. The long-distance obstacle detection method based on image and radar fusion according to claim 4, characterized in that: In step S5, the specific processing flow of performing distortion correction on the image data includes: A special calibration plate is used to take photos, and the world coordinate system is converted to the camera coordinate system using corner point detection, that is, ； The radial distortion coefficient and tangential distortion coefficient are calculated by Taylor formula expansion and the internal parameter matrix is ​​obtained by reconstructing the contrast error. Set the distance offset based on the values ​​of the internal and external parameter matrices; In step S6, the specific process of performing target detection on the image data and assigning different pixel widths and pixel heights to different types of targets includes: using a deep learning network model to detect the target and obtain the type name, and assigning pixel width and height based on prior information to a specific type; In step S7, the specific processing flow of estimating the target distance from the image data includes: Multiply the pixel width and height of the target detected by the deep learning network model by the focal length, divide them by the set pixel width and height respectively, and add the distance offset. Compare the minimum values ​​of the width position and the height position, and use the minimum value as the estimated distance of the target.

9. The long-distance obstacle detection method based on image and radar fusion according to claim 7, characterized in that: In step S8, based on the fusion of image data, point cloud data and inertial navigation data, the specific processing process of determining the validity of the obstacle detection result includes: based on the image data, point cloud data and inertial navigation data, the type and distance information of the obstacle are obtained through the image estimation target distance algorithm and the image and radar fusion algorithm, and according to the constraint condition of the train running speed, the result exceeding the running speed is invalid; In step S9, based on the validity of the obstacle detection result, the specific process of determining the type of dangerous obstacle includes: Analyze the effectiveness of obstacles based on the train speed constraints, and analyze whether the detected obstacles are dangerous obstacles based on the obstacle detection scenarios and types of urban rail trains, and output the information of dangerous obstacles; In step S10, based on the validity of the obstacle detection result, the specific processing process of determining the type of dangerous obstacle includes: based on the determination information of the dangerous obstacle, comparing whether the distance of the obstacle is within the safe distance. If it is within the safe distance, only voice prompt is given; otherwise, an alarm is given and a reminder is given whether to brake.

Images