A railway emergency monitoring method based on radar and visual fusion

Abstract

The present invention discloses a railway emergency monitoring method based on radar-visual fusion, including: joint calibration of radar and camera; radar obtains point cloud data of the current frame of the monitoring area to determine whether there is an obstacle; if so, the first category and first confidence of the obstacle are given; the camera loads calibration parameters according to the obstacle position information in the point cloud data, and shoots the obstacle; determines whether there is an obstacle according to the image data; if so, the second category and second confidence of the obstacle are given; determines whether the first category and the second category are consistent, if so, outputs obstacle data, if not, compares the first confidence and the second confidence, and outputs obstacle data corresponding to the larger one. The present invention can perform intelligent detection of intruders invading the line limit range in various complex and harsh operating environments, and provide classified alarm output, while ensuring the practicality of the product function, it also has a good user experience.

Description

Technical Field

[0001] The present invention relates to the field of rail transit safety technology, and in particular to a railway emergency monitoring method based on radar and visual fusion. Background Art

[0002] my country has a vast territory, long railway lines, and very complex natural and cultural conditions along the lines. As the most sustainable mode of transportation, railways are the main artery of my country's economic development, as well as key infrastructure and important basic industries. The railway emergency monitoring system monitors targets that invade railway lines, automatically identifies targets that endanger driving safety, and records the intrusion behavior or process to ensure railway safety.

[0003] Traditional emergency monitoring technology can no longer meet the requirements of the China Railway Corporation for zero missed reports, low false alarms, and all-weather monitoring. The China Railway Corporation requires emergency monitoring systems to have zero missed reports, low false alarms, and be able to achieve all-weather automatic monitoring. The products currently used on railway operating lines cannot meet the above requirements.

[0004] At present, the products used for railway perimeter intrusion mainly include three types of products: vibration optical fiber, electronic fence, and infrared counter-shooting. All of the above products have problems such as high false alarm rate, inability to judge the behavior of the target after the intrusion, and inability to take into account the monitoring of the railway line.

[0005] The dual power grid uses physical protection + power grid detection to identify foreign object intrusion, which can only be applied to scenes such as highway-railway crossings and bridges. It has the problems of high construction cost, difficult maintenance, and possible underreporting of thrown objects. In addition, most of the system has been in service for more than ten years, and some areas even need to stop driving for maintenance, which has extremely high maintenance costs.

[0006] In recent years, radar technology and intelligent video technology have begun to be applied to crisis monitoring, but the application of a single technology still cannot solve the existing problems. The advantages and disadvantages of radar and video are shown in the following table:

[0007]

[0008] On the other hand, there are also devices and applications of radar-visual fusion monitoring technology that combines radar and video, such as the patent with publication number CN109164443A, which discloses a railway line foreign body detection method based on radar and image analysis, which mainly discloses using radar to monitor whether there are moving targets within the monitoring range, and then further identifying and processing the moving targets through a high-definition pan-tilt camera. It still has the following shortcomings:

[0009] (1) There is no linkage between the microwave radar and the high-definition PTZ camera. The conclusion is drawn after the two devices work independently.

[0010] (2) Its foreign object recognition relies unilaterally on the real-time images captured by the high-definition pan-tilt camera. As mentioned above, when in complex environments (such as rain, snow, and fog) and at night, its recognition accuracy will inevitably be greatly affected, and it may even be impossible to recognize.

[0011] (3) It relies on unilateral foreign body identification and does not have any approved operations, which will inevitably lead to a high rate of false alarms and missed alarms. Summary of the invention

[0012] In order to solve the deficiencies existing in the above-mentioned prior art, the present invention provides a railway danger monitoring method based on radar and video fusion, which better integrates radar and video equipment to form a better linkage and mutual calibration relationship, and can intelligently detect intrusive objects within the line limit range under various complex and harsh operating environments, and provide classified alarm output.

[0013] In order to achieve the above invention object, the technical solution of the present invention is as follows:

[0014] A railway emergency monitoring method based on radar and visual fusion, comprising the steps of:

[0015] Step S1. Jointly calibrate the radar and the camera to obtain calibration parameters of the radar and the camera in different postures;

[0016] Step S2. The radar obtains point cloud data of the current frame of the monitoring area to determine whether there is an obstacle; if so, a first category and a first confidence level of the obstacle are given;

[0017] Step S3. The camera loads the calibration parameters according to the obstacle position information in the point cloud data to adjust the camera posture, photograph the obstacle, and obtain image data of the obstacle;

[0018] Step S4. Determine whether there is an obstacle based on the image data; if so, give a second category and a second confidence level of the obstacle;

[0019] Step S5. Determine whether the first category and the second category are consistent. If so, output the obstacle data. If not, compare the first confidence level and the second confidence level, and output the obstacle data corresponding to the larger one.

[0020] In some preferred embodiments, the calibration parameters in step S1 include: the rotation and translation of the camera relative to the radar.

[0021] In some preferred embodiments, the method of obtaining calibration parameters of the radar and the camera in different postures in step S1 includes:

[0022] Step S101. Establish a transformation matrix M to map the point cloud data (x, y, z) into image data (u, v):

[0023]

[0024] Among them, f u 、f v are the effective focal lengths of the camera in the horizontal and vertical directions respectively; u0 and v0 are the coordinates of the camera principal point; R is the camera rotation matrix; t is the translation vector of the camera relative to the radar;

[0025] Step S102: according to the calibration plane of the camera in different postures, solve the above conversion matrix and obtain R and t as calibration parameters.

[0026] In some preferred embodiments, the step S4 further includes: if not, taking the determination result of the radar as output.

[0027] In some preferred embodiments, step S5 further includes, before outputting the obstacle data:

[0028] The above steps are respectively performed on the subsequent frames of the monitoring area to determine whether the same obstacle exists in the multiple frames at the same time. If so, continue with the subsequent steps; if not, stop the process and determine that there is no obstacle at this time.

[0029] In some preferred embodiments, the radar in step S2 acquires point cloud data of the current frame of the monitoring area, and the method for determining whether there is an obstacle includes:

[0030] Taking a frame without obstacles as a standard frame, and extracting a difference point cloud between the current frame and the standard frame;

[0031] If the difference point cloud is empty, it is determined that there is no obstacle and the process ends;

[0032] If the difference point cloud is not empty, it is determined whether the point cloud is a train; if so, it is determined that there is no obstacle and the process ends; if not, it is determined that there is an obstacle and the process continues.

[0033] Beneficial Effects

[0034] The present invention integrates radar and video systems. With the support of big data and machine learning classification algorithms, it can intelligently detect intruders within the line limit range in various complex and harsh operating environments and provide classified alarm outputs;

[0035] The multiple target recognition and calibration methods of radar and vision fusion can classify intruders and improve the accuracy of recognition; the use of multi-frame comparison, automatic correction calibration, radar and vision integrated coordinate system calibration and other methods increase the system's ability to resist external interference, truly achieving radar and vision integration, zero missed reports, low false alarms, and all-weather guarantee of railway driving safety.

[0036] The system can also provide users with a variety of remote analysis methods such as on-site photos, alarm video clips, point cloud views, etc., providing the necessary conditions for remote alarm confirmation, greatly reducing on-site construction costs, subsequent maintenance costs, manual analysis costs, etc., allowing users to not only monitor the on-site situation in real time around the clock, but also make judgments about the on-site situation clearer and more intuitive. While ensuring the practicality of the product functions, it also has a good user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of a method flow in a preferred embodiment of the present invention;

[0038] Figure 2 A schematic diagram of realizing joint calibration of radar and camera in another preferred embodiment of the present invention; DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings. In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention.

[0040] The present invention provides a railway emergency monitoring method based on radar and visual fusion, such as Figure 1 As shown, the steps include:

[0041] Step S1. Jointly calibrate the radar and the camera to obtain calibration parameters of the radar and the camera in different postures;

[0042] The joint calibration refers to a part of the radar and camera interconnection system calibration, and the system calibration also includes: 1. Monitoring area demarcation; 2. Track plane equation calculation; 3. Camera internal parameter calibration. The determination of these three calibration parameters belongs to the common technology in this field, so the present invention will not repeat them. The joint calibration of radar and camera mainly refers to, Figure 2As shown, the attitude change parameters of the camera relative to the radar under different application states are obtained, preferably the rotation and translation of the camera relative to the radar. The purpose of joint calibration is that when the radar obtains obstacle information, the camera can be quickly positioned to the obstacle material based on the obstacle position information and the mapping relationship between the radar and the camera. Compared with the coordinate transformation and complex calculation methods often used in the prior art, the positioning after joint calibration is faster and saves the computing power resources of related equipment. At the same time, it also avoids the error problem caused by unreasonable positioning algorithm settings.

[0043] In some preferred embodiments, a method for obtaining calibration parameters of radar and camera in different postures is provided, such as Figure 2 As shown, specifically including:

[0044] Step S101. Establish a conversion matrix M to map the three-dimensional point cloud data (x, y, z) acquired by the radar into two-dimensional image data (u, v) acquired by the camera:

[0045]

[0046] Among them, f u 、f v are the effective focal lengths of the camera in the horizontal and vertical directions respectively; u0 and v0 are the coordinates of the camera principal point; R is the camera rotation matrix; t is the translation vector of the camera relative to the radar;

[0047] Step S102: Based on the calibration plane of the camera in different postures, a series of linear equations can be obtained to solve the above conversion matrix and obtain R and t as calibration parameters. The process of obtaining R and t is joint calibration.

[0048] Step S2. The radar obtains point cloud data of the current frame of the monitoring area to determine whether there is an obstacle; if so, a first category and a first confidence level of the obstacle are given;

[0049] Step S3. The camera loads the calibration parameters according to the obstacle position information in the point cloud data to adjust the camera posture, photograph the obstacle, and obtain image data of the obstacle;

[0050] It should be understood that there are many ways to analyze whether there is an obstacle based on point cloud data, such as threshold method, machine learning method, etc., and there are many studies in this area. In some preferred embodiments, a method for determining whether there is an obstacle is given:

[0051] Taking a frame without obstacles as a standard frame, and extracting a difference point cloud between the current frame and the standard frame;

[0052] If the difference point cloud is empty, it is determined that there is no obstacle and the process ends;

[0053] If the difference point cloud is not empty, it is determined whether the point cloud is a train; if so, it is determined that there is no obstacle and the process ends; if not, it is determined that there is an obstacle and the process continues.

[0054] Here is a specific example of this method:

[0055] First, the current frame of the monitoring area is extracted, and the KNN algorithm is used to calculate the point distance difference between the current frame and the calibration frame, and the distance from the point with large difference to the track plane is calculated. The height threshold is used to filter out the points above a certain height of the track plane, and the points above the track plane are recorded as AbovePoints.

[0056] Further determine whether AbovePoints is empty. If it is empty, it is considered that the point cloud of this frame has neither obstacles nor trains. If it is not empty, the height and number of the point cloud are used to determine whether it is a train. If it is a train, the obstacle category of the frame is changed to a train, and the situation of N consecutive frames is calculated to deduce the start and end time of the train, and at the same time, the camera video corresponding to this frame (30 seconds forward from the time when the train is found to the end of the train) is output (the frame of the picture between the start and end of the train is also output separately); if it is not a train, the point cloud is clustered and the point cloud group with a height and number greater than a certain value is selected, that is, the possible obstacle position. Combined with the VFH features of the existing point clouds of various obstacles, the KNN algorithm is used to find the most likely category and calculate the confidence coefficient.

[0057] In particular, in order to further increase the correlation between the image acquired by the camera and the obstacle and increase the recognizability of the obstacle image, the camera posture (including rotation, translation and other operations) is adjusted according to the joint calibration relationship between the camera and the radar and the position coordinates of the obstacle in the point cloud data, and the focal length is adjusted to take a close-up photo of the obstacle, and then the extended area of ​​the obstacle in the picture is deducted according to the joint calibration relationship. It should be noted that the adjustment amount of the camera focal length in this application is part of the joint calibration parameters, which is also mapped to the calibration coordinates, which is different from the commonly used technology in this field. This field generally adopts a simple zoom operation to make the target size in the picture appropriate. At this time, the specific value of the zoom amount depends on manual or other additional algorithms. The disadvantage is that on the one hand, it requires additional operations or computing resources, and on the other hand, the quality of the algorithm determines the quality of the zoom result.

[0058] Step S4. Determine whether there is an obstacle based on the image data; if so, give the second category and second confidence level of the obstacle; in other preferred embodiments, if the current frame is acquired at night, or the environmental conditions during acquisition are relatively bad (such as rain, snow, or foggy weather), the judgment based on the image data may be wrong, such as misjudging an existing obstacle as not existing. Therefore, consider adding a verification step to this step, that is, when it is determined that there is no obstacle based on the image data, the determination result of the radar is used as the output.

[0059] Step S5. Determine whether the first category and the second category are consistent. If so, output the obstacle data. If not, compare the first confidence level and the second confidence level, and output the obstacle data corresponding to the larger one. It should be understood that in other preferred embodiments, when the obstacle categories determined by the two are inconsistent, in order to distinguish the recognition reliability of the radar and the camera in an environment where each has an advantage, consider setting different thresholds according to different external environments to make separate judgments on the two. For example, establish a threshold selection table as shown in Table 1 below. When both meet the threshold, compare which one has a larger value, and the one with a larger confidence value is the one. If only one of them meets the threshold, then that category is output as the result.

[0060] daytime rain Fog, snow Low light Backlight night Radar confidence threshold 0.5 0.6 0.6 0.4 0.25 0.3 Video confidence threshold 0.5 0.4 0.4 0.6 0.75 0.7

[0061] It should be understood that since there may be noise in the surrounding environment, obstacle judgment based on a single frame may have a large error. Therefore, in other preferred embodiments, before outputting the obstacle data, the following steps are further included:

[0062] The above steps are respectively performed on the subsequent frames of the monitoring area to determine whether the same obstacle exists in the multiple frames at the same time. If so, continue with the subsequent steps; if not, stop the process and determine that there is no obstacle at this time.

[0063] It should be understood that after obtaining the obstacle information, the specific obstacle data to be output can be determined by those skilled in the art according to the needs of subsequent systems or the requirements of relevant rules and regulations. In some preferred embodiments, the obstacle data specifically includes: point cloud data, coordinates, size, category, a picture corresponding to the first frame detected, and at least one of the 30 seconds of video before and after the frame image.

[0064] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A railway emergency monitoring method based on radar and visual fusion, characterized in that: Includes steps: Step S1. Jointly calibrate the radar and the camera to obtain calibration parameters of the radar and the camera in different postures; Step S2. The radar obtains point cloud data of the current frame of the monitoring area to determine whether there is an obstacle; if so, a first category and a first confidence level of the obstacle are given; Step S3. The camera loads the calibration parameters according to the obstacle position information in the point cloud data to adjust the camera posture, photograph the obstacle, and obtain image data of the obstacle; Step S4. Determine whether there is an obstacle based on the image data; If yes, then giving a second category and a second confidence level of the obstacle; Step S5. Determine whether the first category and the second category are consistent. If so, output the obstacle data. If not, compare the first confidence level and the second confidence level, and output the obstacle data corresponding to the larger one.

2. The railway emergency monitoring method based on radar and visual fusion as claimed in claim 1, characterized in that: The calibration parameters in step S1 include: the rotation amount and translation amount of the camera relative to the radar.

3. The railway emergency monitoring method based on radar and visual fusion as claimed in claim 2, characterized in that: The method of obtaining calibration parameters of the radar and the camera in different postures in step S1 includes: Step S101. Establish a transformation matrix M to map the point cloud data (x, y, z) into image data (u, v): Among them, f u 、f v are the effective focal lengths of the camera in the horizontal and vertical directions respectively; u0 and v0 are the coordinates of the camera principal point; R is the camera rotation matrix; t is the translation vector of the camera relative to the radar; Step S102: according to the calibration plane of the camera in different postures, solve the above conversion matrix and obtain R and t as calibration parameters.

4. The railway emergency monitoring method based on radar and visual fusion as claimed in claim 1, characterized in that: The step S4 further includes: if not, taking the determination result of the radar as output.

5. The railway emergency monitoring method based on radar and visual fusion as claimed in claim 1, characterized in that: Step S5 also includes, before outputting the obstacle data: The above steps are respectively performed on the subsequent frames of the monitoring area to determine whether the same obstacle exists in the multiple frames at the same time. If so, continue with the subsequent steps; if not, stop the process and determine that there is no obstacle at this time.

6. The railway emergency monitoring method based on radar and visual fusion as claimed in claim 1, characterized in that: In step S2, the radar obtains the point cloud data of the current frame of the monitoring area, and the method for determining whether there is an obstacle includes: Taking a frame without obstacles as a standard frame, and extracting a difference point cloud between the current frame and the standard frame; If the difference point cloud is empty, it is determined that there is no obstacle and the process ends; If the difference point cloud is not empty, it is determined whether the point cloud is a train; if so, it is determined that there is no obstacle and the process ends; if not, it is determined that there is an obstacle and the process continues.

Images