High-speed rail perimeter intrusion alarm method and system based on thunder-vision fusion

Abstract

The invention discloses a high-speed rail perimeter intrusion alarm method and system based on thunder-vision fusion, and solves the problem of low reliability of high-speed rail perimeter intrusion alarm in the prior art, and the method comprises the steps: carrying out the weather detection and rail surface foreign matter detection based on radar data, and generating radar weather information and radar alarm information; weather detection and rail surface foreign matter detection are carried out based on video data, object tracking is carried out on a moving target and a low-confidence target by adopting a similar SORT algorithm in combination with IOU, gradient and LBP, and video weather information and video alarm information are generated; decision-level fusion is carried out on the weather information and the alarm information, and effective alarm information is reserved; and performing priority weighting on the effective alarm information in a period of time, and taking the alarm information with the highest priority as final alarm information. According to the method, the reliability of object tracking is improved, the considered influence factors are more comprehensive, and the reliability of high-speed rail perimeter intrusion alarm is further improved.

Description

Technical Field

[0001] The present invention relates to the field of sensor detection technology, and specifically provides a high-speed railway perimeter intrusion alarm method and system based on radar and vision fusion. Background Art

[0002] In the intrusion alarm of the high-speed rail perimeter, the pure lidar solution is easily interfered by weather factors, which will significantly reduce its performance. For example, heavy rain and dense fog will scatter the laser beam, resulting in reduced ranging accuracy or complete failure. The pure video solution is greatly affected by ambient light. The camera is sensitive to lighting conditions, which affects the clarity and quality of the image, and thus affects the performance of the perception system. In addition, it is difficult to obtain three-dimensional information and cannot accurately locate the position of intruding foreign objects.

[0003] In the existing multi-sensor fusion schemes, there are usually three fusion methods: data-level fusion, feature-level fusion and decision-level fusion. However, data-level fusion and feature-level fusion are the fusion of data information or features from radar and image modalities. Although these methods improve the detection performance compared with pure visual methods, they rely on raw data to a large extent and take a long time to process raw data, resulting in poor real-time performance. The influencing factors considered in the existing decision-level fusion process are not comprehensive, resulting in low reliability of high-speed rail perimeter intrusion alarms. In addition, in the existing radar-vision fusion scheme, the tracking method used by the visual sensor to detect moving targets and low-confidence targets has low reliability, which further affects the reliability of high-speed rail perimeter intrusion alarms. Summary of the invention

[0004] In order to solve the problem of low reliability of high-speed railway perimeter intrusion alarm in the prior art, the present invention provides a high-speed railway perimeter intrusion alarm method and system based on radar and vision fusion, which improves the reliability of object tracking, considers the influencing factors more comprehensively, and further improves the reliability of high-speed railway perimeter intrusion alarm.

[0005] The specific scheme of the present invention is as follows.

[0006] In the first aspect, a high-speed railway perimeter intrusion alarm method based on radar and visual fusion includes: S1: Perform weather detection and track foreign object detection based on radar data, and generate radar weather information and radar alarm information; S2: Perform weather detection and track foreign object detection based on video data. For moving targets and low-confidence targets, a SORT-like algorithm combined with IOU, gradient, and LBP is used to track objects, generating video weather information and video alarm information. S3: Perform decision-level fusion on weather information and alarm information respectively, and retain valid alarm information; S4: Priority weighting is performed on valid alarm information within a period of time, and the alarm information with the highest priority is used as the final alarm information.

[0007] The radar weather information and video weather information include the current weather conditions, such as rainy days, foggy days and other weather conditions, and the probability of each weather condition being identified. The radar alarm information and video alarm information include the coordinates, size and movement speed of foreign objects on the track surface. In step S3, the radar weather information and video weather information are first fused at the decision level, and then the radar alarm information and video alarm information are fused at the decision level according to the fused weather information.

[0008] The present invention adopts a SORT-like algorithm combined with IOU, gradient and LBP to track moving targets and low-confidence targets, enhances the foreign object detection performance of the video, and takes into account multiple influencing factors, so that the video alarm information is more accurate compared with the prior art, further improving the reliability of the high-speed rail perimeter intrusion alarm.

[0009] Furthermore, weather detection based on radar data includes: Divide the point cloud data set in the radar data into a number of voxels, calculate the density of each voxel, generate the density distribution of the point cloud data set, and then calculate the distribution of fog clusters in the three-dimensional space by density clustering. If randomly distributed point cloud clusters appear in the space and the number is greater than a first threshold, it is determined to be foggy weather; The number and distribution of point clouds with reflectivity values ​​lower than the second threshold are calculated and counted. If the point clouds are mainly concentrated within the range close to the radar and within the critical surface of the ground, it is determined to be heavy rain weather.

[0010] In the case of heavy fog, the density of fog in the air increases, the laser radar reflection point cloud is unevenly distributed, and appears in large clusters, so the above method based on voxel density calculation is proposed to determine whether it is heavy fog. Compared with heavy fog, the density of heavy rain is not as high as that of heavy fog. The area within a certain range near the radar installation position is a serious area where heavy rain attenuates the laser beam. At the same time, the critical surface position of the ground contact is also a serious area where the laser beam is reflected and attenuated, so whether it is heavy rain weather is determined based on the distribution of point clouds.

[0011] Furthermore, the detection of foreign objects on the rail surface based on radar data includes: S111: Filter the noise points in the point cloud of radar data, cut the defense zone, filter the rain and fog, and perform offset registration operations; S112: performing elevation restriction fitting on the background point cloud of the air scene to obtain the track plane and the ground surface where the track is located, and performing point cloud segmentation using the track plane and the track surface respectively, so as to segment the point cloud above the ground; S113: Density clustering is performed on the segmented point cloud to form different point cloud clusters, each cluster represents a suspected intrusion target; S114: Kalman filter tracking and Hungarian ID matching are performed on the point cloud clusters frame by frame, and an intrusion alarm ID is output. The process of one frame is terminated, and the calculation of subsequent frames is continued.

[0012] Furthermore, when performing weather detection based on video data, a classification network is used to identify the current weather conditions; detecting foreign objects on the track surface based on video data includes: using a target detection model, training on intruding foreign objects in the track surface scene, using a lower confidence level during reasoning, outputting the target category and location size of the foreign object, and detecting moving foreground targets through background modeling.

[0013] When detecting foreign objects on the rail surface based on video data, the target detection model and background modeling are used to actively identify foreign objects on the rail surface. When training the target detection model, the intruding foreign objects include people, fallen rocks, boxes, branches, woven bags, etc. When performing weather detection, it is necessary to identify rainy days, foggy days and other weather conditions.

[0014] Furthermore, the SORT-like algorithm described in step S2 is combined with IOU, gradient and LBP to track the object, specifically including: Initialize the array, use the trajectory predicted by the Kalman filter to generate a prediction box, calculate the IOU, gradient and LBP of the prediction box and the detection box respectively, use the scharr operator to calculate the gradient, and then weight the IOU, gradient and LBP to generate the matching degree of the prediction box and the detection box. Determine whether the prediction is successful based on the matching degree. If the prediction is successful, the tracking ID remains unchanged. If the prediction fails, a new ID is generated.

[0015] When the scharr operator is used to calculate the gradient, the gradient is calculated in the horizontal and vertical directions respectively, and then the final gradient is calculated; the feature similarity calculated by the SORT-like algorithm includes IOU, gradient and LBP, and is weighted to generate the matching degree. In the weighting process, different weights are selected according to different weather conditions. When the matching degree is greater than or equal to 0.8, it is considered that the prediction box and the detection box match successfully, that is, the prediction is successful. When the matching degree is less than 0.8, it is considered that the prediction box and the detection box do not match, that is, the prediction fails. The present invention uses the above method to track moving targets and low-confidence targets, enhances the foreign body detection performance of the video, and takes into account a variety of influencing factors, so that the video alarm information is more accurate than the prior art, and further improves the reliability of the high-speed rail perimeter intrusion alarm.

[0016] Furthermore, the decision-level fusion of weather information in step S3 includes: weighted averaging the foggy probability, rainy probability, and interference-free weather probability included in the radar weather information and the video weather information, and taking the weather with the highest weighted probability as the current weather condition. Appropriate weights are selected according to the different effects of different weather conditions on the lidar and the camera.

[0017] Furthermore, when the weather condition is foggy, the three-dimensional coordinates of the radar alarm information are mapped to the two-dimensional coordinates of the video data, a mask of the radar alarm information on the two-dimensional image is generated, and the image sequence of the corresponding time period is searched according to the time of the radar alarm. For these image sequences, the target detection neural network is used to perform low-confidence reasoning to generate a bounding box, and the position difference between the mask and the bounding box is calculated, and whether the current alarm is valid is determined based on the position difference.

[0018] When it is considered foggy, the laser radar has limited ability to see through foggy weather and is easily disturbed and causes false alarms, so a more stringent filtering method is used for radar alarm information. The three-dimensional coordinates of the radar alarm information are mapped to the two-dimensional coordinates of the video data to realize the spatial coordinate system of the laser radar and the camera. The specific method is to calculate the intrinsic parameter matrix of the camera video, establish 3D-2D feature point pairs, obtain the transformation matrix between the laser radar coordinate system and the camera coordinate system, and finally obtain the mask of the radar alarm information on the two-dimensional image.

[0019] Furthermore, when the weather condition is rainy, a raindrop filter is added to the video alarm information, specifically including: using a neural network target detection model to identify raindrops, generating a detection trajectory, and collecting the real trajectory of raindrops during heavy rain, constructing a cumulative distance matrix of the detection trajectory and the real trajectory, generating a similarity, and judging whether the current alarm is valid based on the similarity.

[0020] When it is considered to be rainy weather, raindrops and raindrops can easily interfere with video recognition and cause false alarms, so a raindrop filter is added to the video alarm information, mainly through feature matching and motion trajectory matching. When the similarity is greater than the threshold, it is considered to be raindrop interference, and the current alarm is judged as an invalid alarm and filtered.

[0021] Furthermore, the video alarm information is divided into video high-confidence alarm information, video low-confidence alarm information and motion alarm information. The radar alarm information, video high-confidence alarm information and video low-confidence alarm information are weighted in priority, and the alarm information with the highest priority is output; if the valid alarm information described in step S4 only includes motion alarm information, the motion alarm information is output.

[0022] The number of each alarm within a period of time is recorded, and the alarm number is weighted by priority. The priority is from high to low: radar alarm information, video high confidence alarm information, video low confidence alarm information and motion alarm information.

[0023] In the second aspect, a high-speed railway perimeter intrusion alarm system based on radar and visual fusion includes: Radar data processing unit, which performs weather detection and track foreign object detection based on radar data, and outputs radar weather information and radar alarm information to the fusion unit; The video data processing unit performs weather detection and track foreign object detection based on the video data, and outputs video weather information and video alarm information to the fusion unit; The fusion unit performs decision-level fusion on weather information and alarm information, retains valid alarm information and outputs it to the decision unit; The decision-making unit weights the priority of the valid alarm information received within a period of time and outputs the alarm information with the highest priority; the system executes the above-mentioned high-speed railway perimeter intrusion alarm method based on radar and visual fusion.

[0024] In a preferred embodiment, the radar data is generated by a lidar and the video data is generated by a camera.

[0025] Therefore, the present invention has the following beneficial effects: The present invention uses a SORT-like algorithm combined with IOU and gradient and LBP to track moving targets and low-confidence targets, enhances the foreign object detection performance of the video, and considers a variety of influencing factors, so that the video alarm information is more accurate than the prior art, and further improves the reliability of the high-speed rail perimeter intrusion alarm. In addition, the present invention also comprehensively considers a variety of influencing factors in the detection process, and the decision-level fusion process is more perfect, reducing the impact of different weather conditions, making the error after fusion smaller, and further improving the reliability of the high-speed rail perimeter intrusion alarm. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work.

[0027] Figure 1 The present invention is a flow chart of a high-speed railway perimeter intrusion alarm method based on radar and visual fusion.

[0028] Figure 2 It is a structural schematic diagram of a high-speed railway perimeter intrusion alarm system based on radar and vision fusion of the present invention. DETAILED DESCRIPTION

[0029] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0030] like Figure 1 FIG. 1 is a flow chart of a high-speed railway perimeter intrusion alarm method based on radar and visual fusion according to the present invention. Figure 2 Shown is a structural schematic diagram of a high-speed railway perimeter intrusion alarm system based on radar and visual fusion according to the present invention.

[0031] The present invention proposes a high-speed rail perimeter intrusion alarm method based on radar-visual fusion. First, radar data is obtained through a laser radar, and video data is obtained through a camera at the same time, and then weather detection and track surface foreign object detection are performed according to the radar data and the video data respectively. Radar weather information is generated after weather detection is performed according to the radar data, and radar alarm information is generated after track surface foreign object detection is performed; video weather information is generated after weather detection is performed according to the video data, and video alarm information is generated after track surface foreign object detection is performed. When track surface foreign object detection is performed according to video data, a SORT-like algorithm combined with IOU, gradient and LBP is used to track moving targets and low-confidence targets, so that the video alarm information is more reliable than the prior art. The radar weather information and the video weather information are decision-makingly fused to judge the current weather conditions, and then the radar alarm information and the video alarm information are decision-making-level fused according to the current weather conditions, so as to filter out invalid alarm information and retain valid alarm information. Finally, the valid alarm information within a period of time is weighted by priority, and the alarm information with the highest priority is used as the final alarm information.

[0032] The present invention also proposes a high-speed rail perimeter intrusion alarm system based on radar-visual fusion, which executes the above-mentioned high-speed rail perimeter intrusion alarm method based on radar-visual fusion, including a radar data processing unit, a video data processing unit, a fusion unit and a decision unit. First, the laser radar generates radar data, and the camera generates video data. The radar data processing unit performs weather detection and foreign object detection on the track surface based on the radar data, and outputs radar weather information and radar alarm information to the fusion unit; at the same time, the video data processing unit performs weather detection and foreign object detection on the track surface based on the video data, and outputs video weather information and video alarm information to the fusion unit; the fusion unit first performs decision-making fusion on the radar weather information and the video weather information, judges the current weather conditions, and then performs decision-level fusion on the radar alarm information and the video alarm information according to the current weather conditions, thereby filtering out invalid alarm information, retaining valid alarm information and outputting it to the decision unit; the decision unit weights the priority of the valid alarm information received within a period of time, and outputs the alarm information with the highest priority.

[0033] Radar weather information and video weather information include the current weather conditions. The radar data processing unit and the video data processing unit need to identify rainy days, foggy days and other weather conditions, and generate the probability of each identified weather condition. Radar alarm information and video alarm information include information such as the coordinates, size and movement speed of foreign objects on the track surface. The radar data processing unit and the video data processing unit actively identify foreign objects on the track surface. The video data processing unit uses a SORT-like algorithm combined with IOU, gradient and LBP to track moving targets and low-confidence targets, enhances the foreign object detection performance of the video, and considers a variety of influencing factors, so that the video alarm information is more accurate than the existing technology, and further improves the reliability of the high-speed rail perimeter intrusion alarm. In step S3, the radar weather information and the video weather information are first fused at the decision level, and then the radar alarm information and the video alarm information are fused at the decision level according to the fused weather information.

[0034] In a preferred embodiment, weather detection based on radar data includes heavy fog recognition and heavy rain recognition.

[0035] In the case of heavy fog, the density of fog in the air increases, the LiDAR reflection point cloud is unevenly distributed, and appears in large clusters. Therefore, a method based on voxel density calculation is proposed for heavy fog recognition to determine whether it is foggy weather. The specific steps are as follows: S101: Divide the point cloud data set in the radar data into a number of small voxels, where a voxel is a small square in a three-dimensional space; S102: Calculate the number of point clouds in each voxel, and then obtain the density of each voxel; S103: Calculate the voxel density of the entire point cloud data set to obtain the density distribution of the entire point cloud data set; S104: Calculate the distribution of fog clusters in the three-dimensional space by density clustering. If randomly distributed point cloud clusters appear in the space and the number is greater than the first threshold, it is determined that the current weather condition is foggy.

[0036] Compared with heavy fog, the density of heavy rain is not as high as that of heavy fog. The area within a certain range close to the installation position of the lidar is a serious area where heavy rain causes attenuation of the laser beam. At the same time, the critical surface of contact with the ground is also a serious area where the laser beam produces reflection and attenuation. Therefore, whether it is heavy rain weather is determined based on the distribution of point clouds. The specific method is as follows: calculate the number of point clouds with point cloud reflectivity values ​​lower than the second threshold, and count the distribution of point clouds that meet this condition. If the point clouds are mainly concentrated in the range close to the radar and within the critical surface of contact with the ground, it is determined that the current weather condition is heavy rain weather.

[0037] When the radar data processing unit detects foreign objects on the track surface based on radar data, it first performs filtering processing and defense zone cutting, rain and fog filtering, and offset registration on the noise points in the point cloud in the radar data; then, the background point cloud of the empty scene is fitted with elevation restrictions to obtain the track plane and the ground surface where the track is located, and the point cloud is segmented by the track plane and the track surface respectively, and the point cloud above the ground is segmented; the segmented point cloud is density clustered into different point cloud clusters, each cluster represents a suspected intrusion target; finally, the point cloud clusters are subjected to Kalman filter tracking and Hungarian ID matching frame by frame, and the intrusion alarm ID is output. End the process of one frame, continue the calculation of subsequent frames, and finally generate radar alarm information according to the processing results.

[0038] In another preferred embodiment, the video data processing unit uses a classification network to identify the current weather conditions when performing weather detection based on the video data. When performing weather detection, it is necessary to identify rainy days, foggy days and other weather conditions. When the video data processing unit performs track surface foreign object detection based on the video data, it actively identifies track surface foreign objects through target detection models and background modeling. When using the target detection model, the intruding foreign objects in the track surface scene are trained. The intruding foreign objects include people, fallen rocks, boxes, branches, woven bags, etc. A lower confidence is used during reasoning to output the target category and location size of the foreign object. In addition, the moving foreground target is detected through background modeling.

[0039] The SORT algorithm in the prior art is prone to ID switching in actual scene applications, so this embodiment uses a SORT-like algorithm combined with IOU, gradient, and LBP to track moving targets and low-confidence targets, and makes certain adjustments to only retain targets whose tracking IDs can be retained for more than a third threshold number of frames. The third threshold is usually 5. The specific steps are as follows: S201: Initialize the array, generate a prediction box using the trajectory predicted by the Kalman filter, mark the prediction box as k1, and mark the detection box as k2; S202: Use the Scharr operator to calculate the gradient in the horizontal and vertical directions respectively: Then calculate the final gradient: Here, the gradients of the prediction box k1 and the detection box k2 are calculated respectively; S203: Perform IOU calculation and LBP calculation on the prediction box k1 and the detection box k2 respectively; S204: Weight the calculation results of IOU, gradient and LBP to generate the matching degree of prediction box k1 and detection box k2: F(x)=a*iou+b*G(x)+c*lbp, Among them, a, b, and c take different values ​​according to different weather conditions. When the weather is undisturbed, a is 0.5, b is 0.25, and c is 0.25. When the weather is rainy and foggy, a is 0.2, b is 0.4, and c is 0.4. Then the matching degree is used to determine whether the prediction is successful. When F(x) is greater than or equal to 0.8, it is considered that the prediction box and the detection box match successfully, that is, the prediction is successful, and the tracking id remains unchanged. When F(x) is less than 0.8, it is considered that the prediction box and the detection box do not match, that is, the prediction fails, and a new id is generated.

[0040] The present invention adopts the above method to track moving targets and low-confidence targets, enhances the foreign object detection performance of the video, takes into account multiple influencing factors, thereby making the video alarm information more accurate compared with the prior art, and further improving the reliability of the high-speed rail perimeter intrusion alarm.

[0041] In another preferred embodiment, when the fusion unit performs decision-level fusion on the radar weather information and the video weather information, since both the radar weather information and the video weather information include foggy day probability, rainy day probability and non-interference weather probability, the foggy day probability, rainy day probability and non-interference weather probability included in the radar weather information and the video weather information are weighted averaged, and the weather with the highest probability calculated after weighting is taken as the current weather condition. According to the different influences of different weather conditions on the laser radar and the camera, the appropriate weight is selected.

[0042] The specific method is as follows: the radar weather information output by the radar processing unit includes the radar fog probability x1, the radar rain probability x2, the radar interference-free weather probability x3 and the radar fog group position box1; the video weather information output by the video processing unit includes the video fog probability y1, the video rain probability y2 and the video interference-free weather probability y3; by weighted averaging the fog probability, the rain probability and the interference-free weather probability respectively, we can obtain: P(A1)=a1×x1+b1×y1, P(A2)=a2×x2+b2×y2, P(A3)=a3×x3+b3×y3, P=max(O(A1),P(A2),P(A3)), Among them, a1 is usually set to 0.7, b1 is usually set to 0.3, a2 is usually set to 0.7, b2 is usually set to 0.3, a3 is usually set to 0.9, and b1 is usually set to 0.3; finally, the highest value among P(A1), P(A2), and P(A3) is used as the current weather conditions.

[0043] When the weather condition is judged to be foggy, the laser radar has limited ability to see through foggy days and is easily disturbed and causes false alarms, so a more stringent filtering method is used for radar alarm information. The three-dimensional coordinates of the radar alarm information are mapped to the two-dimensional coordinates of the video data to realize the spatial coordinate system of the laser radar and the camera. The specific method is to calculate the intrinsic parameter matrix of the camera video, and then establish a 3D-2D feature point pair to obtain the transformation matrix between the laser radar coordinate system and the camera coordinate system, and finally obtain the mask of the radar alarm information on the two-dimensional image, recorded as mask1; then search for the image sequence of the corresponding time period according to the time of the radar alarm, which is usually about 10 seconds. For these image sequences, use the target detection neural network for low-confidence reasoning to generate a bounding box, recorded as bbox1; then calculate the position difference between the mask mask1 and the bounding box bbox1, and judge whether the current alarm is valid based on the position difference.

[0044] The position difference between mask mask1 and bounding box bbox1 is calculated as follows: area1 represents the overlapping area of ​​mask1 and bbox1. area2=area1 / mask1, area3 = area1 / bbox1, When area1 is not 0, area2 is greater than 0.2, and area3 is greater than 0.1, it is considered that the radar alarm information and the video alarm information overlap, and the current alarm is determined to be valid.

[0045] When the weather condition is judged to be rainy, raindrops and raindrops can easily interfere with video recognition and cause false alarms, so a raindrop filter is added to the video alarm information, mainly through feature matching and motion trajectory matching. The neural network target detection model is used to identify raindrops, generate detection trajectories, and collect the real trajectories of raindrops during heavy rain. The cumulative distance matrix of the detection trajectory and the real trajectory is constructed to generate similarity. The similarity is used to determine whether the current alarm is valid. When the similarity is greater than the threshold, it is considered to be raindrop interference, and the current alarm is judged as an invalid alarm and filtered.

[0046] The specific steps are as follows: S301: using a neural network target detection model to identify raindrops, and obtaining a detection trajectory after multiple frame recognition; S302: Collecting the real trajectory of raindrops during heavy rain; S303: define the detection trajectory of raindrops as X = {x1, x2, ..., xn}, the collected real trajectory as Y = {y1, y2, ..., ym}, and construct an n×m distance matrix D, where D[i, j] represents the distance between xi and yj, and the Euclidean distance is used here; S304: construct a cumulative distance matrix, and return the minimum distance in the matrix as the similarity of the two trajectories; S305: convert the next motion trajectory and continue with step S303; S306: Output the highest similarity value, and take the threshold as 0.9. When the similarity is greater than the threshold, it is considered to be raindrop interference, and the current alarm is judged to be an invalid alarm, and the alarm is filtered.

[0047] Video alarm information is divided into video high confidence alarm information, video low confidence alarm information and motion alarm information. First, the number of each alarm within a period of time is recorded, and the number of alarms of radar alarm information, video high confidence alarm information and video low confidence alarm information is weighted. The priority is from high to low, radar alarm information, video high confidence alarm information, and video low confidence alarm information. After weighted calculation, the alarm information with the highest priority is output. If the valid alarm information collected by the decision unit only includes motion alarm information, the motion alarm information is output.

[0048] The specific method is as follows: in the valid alarm information received by the decision unit within a period of time, the number of radar alarms is recorded as a, the number of video high-confidence alarm object recognition is recorded as b, the number of video low-confidence alarm object recognition is recorded as c, and the number of motion alarm recognition is recorded as d. After priority weighting, the final probabilities of each are as follows: P(A)=a*(1+α), P(B)=b*(1+β), P(C)=c*(1+γ), Because radar alarm is more reliable in most cases, α is taken as 0.5, β is taken as 0.3, and γ is taken as 0.1. The one with the highest final probability is selected as the alarm result. When P(A), P(B), and P(C) are all 0, if d is not 0, the motion alarm is selected as the alarm result.

[0049] In such Figure 1 In the flowchart of a high-speed rail perimeter intrusion alarm method based on radar-visual fusion of the present invention, the laser radar and the camera output radar data and video data respectively at the same time. The radar data is output to the radar data processing unit to generate three-dimensional alarm coordinates and environmental information, which is radar weather information. The video data is output to the video data processing unit, and video weather information is generated through the weather classification network. The radar weather information and the video weather information are fused at the decision level through the fusion unit. If the obtained interference weather probability meets the threshold, the filtering mode is turned on.

[0050] The radar three-dimensional alarm coordinates are converted into a two-dimensional bbox to determine whether the detected object has speed; if the detected object does not have speed, it is filtered through the filtering mode. At this time, if the filtering mode is not turned on, it is directly output to the alarm decision module. If the filtering mode is turned on, the closest image sequence is matched according to time; if the detected object has speed, the closest image sequence is directly matched according to time. After matching the closest image sequence according to time, the neural network low-confidence recognition and bbox position change detection are performed respectively. If the GIOU of the bbox2 identified by the neural network low-confidence and the radar bbox1 is greater than the threshold, or the bbox position change detection result is a change, it is output to the alarm decision module.

[0051] The video data processing unit detects the foreground target through background modeling, identifies the target category and location size of the foreign object through the neural network target detection model, and generates high-confidence results and low-confidence results. The high-confidence result is directly output to the alarm decision module, and the low-confidence result is combined with the detected foreground target to perform target tracking. If the target tracking frame number meets the requirements, it is output to the alarm decision module. The alarm information output to the alarm decision module is the valid alarm information, and the alarm information not output to the alarm decision module is the invalid alarm information. Finally, the alarm decision module performs priority weighting on the collected valid alarm information, and outputs the alarm information with the highest priority as the alarm information finally output by the high-speed rail perimeter intrusion alarm method and system based on radar-visual fusion of the present invention.

[0052] In the existing multi-sensor fusion scheme, there are usually three fusion modes: data-level fusion, feature-level fusion and decision-level fusion. However, data-level fusion and feature-level fusion are largely dependent on raw data, and it may take a long time to process the raw data, and the data operation leads to poor real-time performance. The camera and lidar decision-level fusion detection scheme adopted by the present invention has good real-time and adaptability, low data requirements, strong anti-interference ability, and good error correction ability. The error caused by a single sensor is eliminated through appropriate fusion to obtain the correct detection result.

[0053] The present invention also uses a SORT-like algorithm combined with IOU and gradient and LBP to track moving targets and low-confidence targets, enhances the foreign object detection performance of the video, and considers a variety of influencing factors, so that the video alarm information is more accurate than the prior art, and further improves the reliability of the high-speed rail perimeter intrusion alarm. In addition, the present invention also comprehensively considers a variety of influencing factors in the detection process, and the decision-level fusion process is more perfect, reducing the impact of different weather conditions, making the error after fusion smaller, and further improving the reliability of the high-speed rail perimeter intrusion alarm.

[0054] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural transformation made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A high-speed railway perimeter intrusion alarm method based on radar and visual fusion, characterized in that: include: S1: Perform weather detection and track foreign object detection based on radar data, and generate radar weather information and radar alarm information; S2: Perform weather detection and track foreign object detection based on video data. For moving targets and low-confidence targets, a SORT-like algorithm combined with IOU, gradient, and LBP is used to track objects, generating video weather information and video alarm information. S3: Perform decision-level fusion on weather information and alarm information respectively, and retain valid alarm information; S4: Priority weighting is performed on valid alarm information within a period of time, and the alarm information with the highest priority is used as the final alarm information.

2. According to the high-speed railway perimeter intrusion alarm method based on radar and visual fusion according to claim 1, it is characterized in that: Weather detection based on radar data includes: Divide the point cloud data set in the radar data into a number of voxels, calculate the density of each voxel, generate the density distribution of the point cloud data set, and then calculate the distribution of fog clusters in the three-dimensional space by density clustering. If randomly distributed point cloud clusters appear in the space and the number is greater than a first threshold, it is determined to be foggy weather; The number and distribution of point clouds with reflectivity values ​​lower than the second threshold are calculated and counted. If the point clouds are mainly concentrated within the range close to the radar and within the critical surface of the ground, it is determined to be heavy rain weather.

3. According to the high-speed railway perimeter intrusion alarm method based on radar and visual fusion according to claim 2, it is characterized in that: Track surface foreign body detection based on radar data includes: S111: Filter the noise points in the point cloud of radar data, cut the defense zone, filter the rain and fog, and perform offset registration operations; S112: performing elevation restriction fitting on the background point cloud of the air scene to obtain the track plane and the ground surface where the track is located, and performing point cloud segmentation using the track plane and the track surface respectively, so as to segment the point cloud above the ground; S113: performing density clustering on the segmented point cloud to divide it into different point cloud clusters, each cluster representing a suspected intrusion target; S114: performing Kalman filter tracking and Hungarian ID matching on the point cloud clusters frame by frame, and outputting an intrusion alarm ID.

4. The high-speed railway perimeter intrusion alarm method based on radar and visual fusion according to claim 1 is characterized in that: When performing weather detection based on video data, a classification network is used to identify the current weather conditions; Track surface foreign object detection based on video data includes: using a target detection model, training on intruding foreign objects in track surface scenes, using a lower confidence level during reasoning, outputting the target category and location size of the foreign objects, and detecting moving foreground targets through background modeling.

5. The high-speed railway perimeter intrusion alarm method based on radar and visual fusion according to claim 4 is characterized in that: The SORT-like algorithm described in step S2 is combined with IOU, gradient and LBP to track the object, specifically including: Initialize the array, use the trajectory predicted by the Kalman filter to generate a prediction box, calculate the IOU, gradient and LBP of the prediction box and the detection box respectively, use the scharr operator to calculate the gradient, and then weight the IOU, gradient and LBP to generate the matching degree of the prediction box and the detection box. Determine whether the prediction is successful based on the matching degree. If the prediction is successful, the tracking ID remains unchanged. If the prediction fails, a new ID is generated.

6. The high-speed railway perimeter intrusion alarm method based on radar and visual fusion according to claim 1 is characterized in that: The decision-level fusion of weather information described in step S3 includes: taking weighted averages of the foggy day probability, rainy day probability and interference-free weather probability included in the radar weather information and the video weather information, and taking the weather with the highest weighted probability as the current weather condition.

7. The high-speed railway perimeter intrusion alarm method based on radar and visual fusion according to claim 6 is characterized in that: When the weather condition is foggy, the three-dimensional coordinates of the radar alarm information are mapped to the two-dimensional coordinates of the video data, a mask of the radar alarm information on the two-dimensional image is generated, and the image sequence of the corresponding time period is found according to the time of the radar alarm. For these image sequences, the target detection neural network is used to perform low-confidence reasoning to generate a bounding box, and the position difference between the mask and the bounding box is calculated. According to the position difference, it is determined whether the current alarm is valid.

8. The high-speed railway perimeter intrusion alarm method based on radar and visual fusion according to claim 7 is characterized in that: When the weather condition is rainy, a raindrop filter is added to the video alarm information, specifically including: using a neural network target detection model to identify raindrops, generating detection trajectories, and collecting real trajectories of raindrops during heavy rain, constructing a cumulative distance matrix of the detection trajectories and real trajectories, generating similarities, and judging whether the current alarm is valid based on the similarities.

9. The high-speed railway perimeter intrusion alarm method based on radar and visual fusion according to claim 8 is characterized in that: The video alarm information is divided into video high-confidence alarm information, video low-confidence alarm information and motion alarm information. The radar alarm information, video high-confidence alarm information and video low-confidence alarm information are weighted in priority, and the alarm information with the highest priority is output; if the valid alarm information described in step S4 only includes motion alarm information, the motion alarm information is output.

10. A high-speed railway perimeter intrusion alarm system based on radar and visual fusion, characterized in that: include: Radar data processing unit, which performs weather detection and track foreign object detection based on radar data, and outputs radar weather information and radar alarm information to the fusion unit; The video data processing unit performs weather detection and track foreign object detection based on the video data, and outputs video weather information and video alarm information to the fusion unit; The fusion unit performs decision-level fusion on weather information and alarm information, retains valid alarm information and outputs it to the decision-making unit; The decision-making unit weights the priority of the valid alarm information received within a period of time and outputs the alarm information with the highest priority; The system executes a high-speed rail perimeter intrusion alarm method based on radar and visual fusion as described in any one of claims 1-9.

Images